2023-11-24T08:15:41.410 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-24T08:15:41.410 DEBUG:teuthology.run:Teuthology command: teuthology --name dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi --verbose --archive /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501 --owner scheduled_dis@teuthology --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} -- /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501/orig.config.yaml 2023-11-24T08:15:41.461 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-24T08:15:41.547 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501 branch: wip-dis-testing description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: idryomov@gmail.com first_in_suite: false job_id: '7465501' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing sha1: 135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 owner: scheduled_dis@teuthology priority: 69 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 2661 sha1: 135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 sleep_before_teardown: 0 subset: 14/99 suite: rbd suite_branch: wip-dis-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: 135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKay/u69gbSDH+wQg1rANAqFcLSHi3oa5yrvt7QTzuCp6Tnk1XI0S9vKeGRUUVuR7yn7Ern95TqUza/7zUGQSdo= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNN4flvWtU6IA2MNQJD2lLFhHIgd/PZ0iehpJcUsE7kQw/8ZZUYwpfu5sODRLPlVm8TccYiZoQyDcPZAMHRMrk= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 2442239d2653456406c25ae0c71b689c8f2657b6 timestamp: 2023-11-24_05:31:39 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-24T08:15:41.547 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa; will attempt to use it 2023-11-24T08:15:41.548 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks 2023-11-24T08:15:41.548 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-24T08:15:41.550 INFO:teuthology.task.internal:Checking packages... 2023-11-24T08:15:41.574 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1' 2023-11-24T08:15:41.574 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-24T08:15:41.574 INFO:teuthology.packaging:ref: None 2023-11-24T08:15:41.574 INFO:teuthology.packaging:tag: None 2023-11-24T08:15:41.574 INFO:teuthology.packaging:branch: wip-dis-testing 2023-11-24T08:15:41.574 INFO:teuthology.packaging:sha1: 135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 2023-11-24T08:15:41.574 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing 2023-11-24T08:15:41.714 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7484-g135a1ac9-1jammy 2023-11-24T08:15:41.714 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-24T08:15:41.721 INFO:teuthology.task.internal:no buildpackages task found 2023-11-24T08:15:41.721 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-24T08:15:41.736 INFO:teuthology.task.internal:Saving configuration 2023-11-24T08:15:41.754 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-24T08:15:41.760 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T08:15:41.783 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 08:10:02.400326', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM6Fdb+BL2WKf1lMzTCIbR0dqfrNCiO5OSQvu7264gdC'} 2023-11-24T08:15:41.808 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 08:10:02.399211', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP6DcRA/d1U7nH0C9XtRw/Dsw5L9QV7OBIIm6pFnmEEx'} 2023-11-24T08:15:41.808 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-24T08:15:41.814 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-11-24T08:15:41.814 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-11-24T08:15:41.814 INFO:teuthology.run_tasks:Running task console_log... 2023-11-24T08:15:41.888 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f423377eee0>, signals=[15]) 2023-11-24T08:15:41.889 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-24T08:15:41.894 INFO:teuthology.task.internal:Opening connections... 2023-11-24T08:15:41.895 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T08:15:41.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:15:41.971 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T08:15:41.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:15:42.047 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-24T08:15:42.053 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2023-11-24T08:15:42.058 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2023-11-24T08:15:42.059 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2023-11-24T08:15:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-24T08:15:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Ubuntu" 2023-11-24T08:15:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="22.04" 2023-11-24T08:15:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-24T08:15:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_CODENAME=jammy 2023-11-24T08:15:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:ID=ubuntu 2023-11-24T08:15:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE=debian 2023-11-24T08:15:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-24T08:15:42.111 INFO:teuthology.orchestra.run.smithi153.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-24T08:15:42.111 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-24T08:15:42.111 INFO:teuthology.orchestra.run.smithi153.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-24T08:15:42.111 INFO:teuthology.orchestra.run.smithi153.stdout:UBUNTU_CODENAME=jammy 2023-11-24T08:15:42.111 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2023-11-24T08:15:42.135 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2023-11-24T08:15:42.140 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2023-11-24T08:15:42.140 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2023-11-24T08:15:42.185 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Ubuntu" 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="22.04" 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_CODENAME=jammy 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:ID=ubuntu 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE=debian 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-24T08:15:42.186 INFO:teuthology.orchestra.run.smithi164.stdout:UBUNTU_CODENAME=jammy 2023-11-24T08:15:42.187 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2023-11-24T08:15:42.210 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-24T08:15:42.220 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-24T08:15:42.225 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-24T08:15:42.226 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2023-11-24T08:15:42.228 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2023-11-24T08:15:42.232 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-24T08:15:42.238 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-24T08:15:42.238 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2023-11-24T08:15:42.275 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2023-11-24T08:15:42.282 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-24T08:15:42.333 INFO:teuthology.run_tasks:Running task kernel... 2023-11-24T08:15:42.349 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-24T08:15:42.350 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-24T08:15:42.350 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-24T08:15:42.351 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-24T08:15:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:15:42.355 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2023-11-24T08:15:42.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:15:42.357 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2023-11-24T08:15:42.402 INFO:teuthology.orchestra.run.smithi164.stdout:5.15.0-56-generic 2023-11-24T08:15:42.402 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.15.0-56-generic 2023-11-24T08:15:42.402 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get clean 2023-11-24T08:15:42.403 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-56-generic 2023-11-24T08:15:42.404 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.15.0-56-generic 2023-11-24T08:15:42.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2023-11-24T08:15:42.538 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get update 2023-11-24T08:15:42.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2023-11-24T08:15:42.687 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-24T08:15:42.794 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-24T08:15:42.857 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-24T08:15:42.859 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-24T08:15:42.873 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [371 kB] 2023-11-24T08:15:42.876 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-24T08:15:42.918 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-24T08:15:42.937 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [985 kB] 2023-11-24T08:15:42.968 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [192 kB] 2023-11-24T08:15:42.972 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-24T08:15:42.972 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-11-24T08:15:42.973 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,134 kB] 2023-11-24T08:15:42.986 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [185 kB] 2023-11-24T08:15:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-11-24T08:15:42.988 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [567 kB] 2023-11-24T08:15:42.993 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [799 kB] 2023-11-24T08:15:42.999 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [148 kB] 2023-11-24T08:15:43.000 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-24T08:15:43.034 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-24T08:15:43.034 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-24T08:15:43.035 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-24T08:15:43.035 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-24T08:15:43.237 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-24T08:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-24T08:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [985 kB] 2023-11-24T08:15:43.375 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,200 kB] 2023-11-24T08:15:43.418 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,200 kB] 2023-11-24T08:15:43.654 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [536 kB] 2023-11-24T08:15:43.668 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [371 kB] 2023-11-24T08:15:43.690 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [192 kB] 2023-11-24T08:15:43.692 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [252 kB] 2023-11-24T08:15:43.698 INFO:teuthology.orchestra.run.smithi164.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [536 kB] 2023-11-24T08:15:43.700 INFO:teuthology.orchestra.run.smithi164.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-24T08:15:43.701 INFO:teuthology.orchestra.run.smithi164.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-11-24T08:15:43.702 INFO:teuthology.orchestra.run.smithi164.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,134 kB] 2023-11-24T08:15:43.709 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-24T08:15:43.710 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,168 kB] 2023-11-24T08:15:43.737 INFO:teuthology.orchestra.run.smithi164.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [252 kB] 2023-11-24T08:15:43.754 INFO:teuthology.orchestra.run.smithi164.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-24T08:15:43.765 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-11-24T08:15:43.766 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [190 kB] 2023-11-24T08:15:43.772 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-11-24T08:15:43.778 INFO:teuthology.orchestra.run.smithi164.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,168 kB] 2023-11-24T08:15:43.788 INFO:teuthology.orchestra.run.smithi164.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [185 kB] 2023-11-24T08:15:43.799 INFO:teuthology.orchestra.run.smithi164.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-11-24T08:15:43.799 INFO:teuthology.orchestra.run.smithi164.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [567 kB] 2023-11-24T08:15:43.809 INFO:teuthology.orchestra.run.smithi164.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-11-24T08:15:43.810 INFO:teuthology.orchestra.run.smithi164.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [190 kB] 2023-11-24T08:15:43.817 INFO:teuthology.orchestra.run.smithi164.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-11-24T08:15:43.826 INFO:teuthology.orchestra.run.smithi164.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [799 kB] 2023-11-24T08:15:43.936 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,003 kB] 2023-11-24T08:15:43.981 INFO:teuthology.orchestra.run.smithi164.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,003 kB] 2023-11-24T08:15:44.014 INFO:teuthology.orchestra.run.smithi164.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [148 kB] 2023-11-24T08:15:44.409 INFO:teuthology.orchestra.run.smithi164.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-24T08:15:44.415 INFO:teuthology.orchestra.run.smithi164.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-24T08:15:44.417 INFO:teuthology.orchestra.run.smithi164.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-24T08:15:44.431 INFO:teuthology.orchestra.run.smithi164.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-24T08:15:44.434 INFO:teuthology.orchestra.run.smithi164.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-24T08:15:44.562 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [667 kB] 2023-11-24T08:15:44.601 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [221 kB] 2023-11-24T08:15:44.607 INFO:teuthology.orchestra.run.smithi164.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [667 kB] 2023-11-24T08:15:44.613 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-11-24T08:15:44.614 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-24T08:15:44.617 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-24T08:15:44.617 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-24T08:15:44.618 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-24T08:15:44.618 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-24T08:15:44.645 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-24T08:15:44.645 INFO:teuthology.orchestra.run.smithi164.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [221 kB] 2023-11-24T08:15:44.648 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-24T08:15:44.658 INFO:teuthology.orchestra.run.smithi164.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-11-24T08:15:44.659 INFO:teuthology.orchestra.run.smithi164.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-24T08:15:44.661 INFO:teuthology.orchestra.run.smithi164.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-24T08:15:44.662 INFO:teuthology.orchestra.run.smithi164.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-24T08:15:44.663 INFO:teuthology.orchestra.run.smithi164.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-24T08:15:44.663 INFO:teuthology.orchestra.run.smithi164.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-24T08:15:44.688 INFO:teuthology.orchestra.run.smithi164.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-24T08:15:44.692 INFO:teuthology.orchestra.run.smithi164.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-24T08:15:44.812 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-24T08:15:44.857 INFO:teuthology.orchestra.run.smithi164.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-24T08:15:44.895 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-24T08:15:44.940 INFO:teuthology.orchestra.run.smithi164.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-24T08:15:44.979 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-24T08:15:45.024 INFO:teuthology.orchestra.run.smithi164.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-24T08:15:45.060 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-24T08:15:45.094 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-24T08:15:45.105 INFO:teuthology.orchestra.run.smithi164.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-24T08:15:45.140 INFO:teuthology.orchestra.run.smithi164.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-24T08:15:49.321 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 10.4 MB in 2s (4,182 kB/s) 2023-11-24T08:15:49.511 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 10.4 MB in 2s (4,177 kB/s) 2023-11-24T08:15:50.573 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-24T08:15:50.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-24T08:15:50.647 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-24T08:15:50.763 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2023-11-24T08:15:50.780 DEBUG:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-24T08:15:50.834 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2023-11-24T08:15:50.835 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2023-11-24T08:15:50.840 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2023-11-24T08:15:50.998 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2023-11-24T08:15:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-24T08:15:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-24T08:15:50.998 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-24T08:15:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-24T08:15:50.999 INFO:teuthology.orchestra.run.smithi153.stdout: python2.7-minimal 2023-11-24T08:15:50.999 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-24T08:15:51.000 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2023-11-24T08:15:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-11-24T08:15:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-generic linux-image-5.15.0-89-generic 2023-11-24T08:15:51.000 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-5.15.0-89-generic linux-modules-extra-5.15.0-89-generic 2023-11-24T08:15:51.001 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2023-11-24T08:15:51.001 INFO:teuthology.orchestra.run.smithi153.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-24T08:15:51.001 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2023-11-24T08:15:51.002 INFO:teuthology.orchestra.run.smithi153.stdout: thermald 2023-11-24T08:15:51.030 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2023-11-24T08:15:51.031 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2023-11-24T08:15:51.064 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2023-11-24T08:15:51.064 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-11-24T08:15:51.064 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.15.0-89-generic linux-modules-5.15.0-89-generic 2023-11-24T08:15:51.065 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-extra-5.15.0-89-generic 2023-11-24T08:15:51.066 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages will be upgraded: 2023-11-24T08:15:51.066 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-24T08:15:51.121 INFO:teuthology.orchestra.run.smithi153.stdout:3 upgraded, 5 newly installed, 0 to remove and 273 not upgraded. 2023-11-24T08:15:51.121 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 113 MB of archives. 2023-11-24T08:15:51.122 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-24T08:15:51.122 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-89-generic amd64 5.15.0-89.99 [22.4 MB] 2023-11-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages were automatically installed and are no longer required: 2023-11-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi164.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi164.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-24T08:15:51.189 INFO:teuthology.orchestra.run.smithi164.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-24T08:15:51.189 INFO:teuthology.orchestra.run.smithi164.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-24T08:15:51.189 INFO:teuthology.orchestra.run.smithi164.stdout: python2.7-minimal 2023-11-24T08:15:51.189 INFO:teuthology.orchestra.run.smithi164.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-24T08:15:51.190 INFO:teuthology.orchestra.run.smithi164.stdout:The following additional packages will be installed: 2023-11-24T08:15:51.190 INFO:teuthology.orchestra.run.smithi164.stdout: linux-generic linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-11-24T08:15:51.190 INFO:teuthology.orchestra.run.smithi164.stdout: linux-headers-generic linux-image-5.15.0-89-generic 2023-11-24T08:15:51.190 INFO:teuthology.orchestra.run.smithi164.stdout: linux-modules-5.15.0-89-generic linux-modules-extra-5.15.0-89-generic 2023-11-24T08:15:51.191 INFO:teuthology.orchestra.run.smithi164.stdout:Suggested packages: 2023-11-24T08:15:51.191 INFO:teuthology.orchestra.run.smithi164.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-24T08:15:51.191 INFO:teuthology.orchestra.run.smithi164.stdout:Recommended packages: 2023-11-24T08:15:51.191 INFO:teuthology.orchestra.run.smithi164.stdout: thermald 2023-11-24T08:15:51.241 INFO:teuthology.orchestra.run.smithi164.stdout:The following NEW packages will be installed: 2023-11-24T08:15:51.241 INFO:teuthology.orchestra.run.smithi164.stdout: linux-headers-5.15.0-89 linux-headers-5.15.0-89-generic 2023-11-24T08:15:51.242 INFO:teuthology.orchestra.run.smithi164.stdout: linux-image-5.15.0-89-generic linux-modules-5.15.0-89-generic 2023-11-24T08:15:51.242 INFO:teuthology.orchestra.run.smithi164.stdout: linux-modules-extra-5.15.0-89-generic 2023-11-24T08:15:51.243 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages will be upgraded: 2023-11-24T08:15:51.243 INFO:teuthology.orchestra.run.smithi164.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-24T08:15:51.330 INFO:teuthology.orchestra.run.smithi164.stdout:3 upgraded, 5 newly installed, 0 to remove and 273 not upgraded. 2023-11-24T08:15:51.331 INFO:teuthology.orchestra.run.smithi164.stdout:Need to get 113 MB of archives. 2023-11-24T08:15:51.331 INFO:teuthology.orchestra.run.smithi164.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-24T08:15:51.331 INFO:teuthology.orchestra.run.smithi164.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-89-generic amd64 5.15.0-89.99 [22.4 MB] 2023-11-24T08:15:51.779 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-89-generic amd64 5.15.0-89.99 [11.5 MB] 2023-11-24T08:15:52.094 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-89-generic amd64 5.15.0-89.99 [63.9 MB] 2023-11-24T08:15:52.232 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-89-generic amd64 5.15.0-89.99 [11.5 MB] 2023-11-24T08:15:52.502 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-89-generic amd64 5.15.0-89.99 [63.9 MB] 2023-11-24T08:15:53.660 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.89.86 [1,698 B] 2023-11-24T08:15:53.660 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.89.86 [2,492 B] 2023-11-24T08:15:53.661 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89 all 5.15.0-89.99 [12.3 MB] 2023-11-24T08:15:54.094 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89-generic amd64 5.15.0-89.99 [2,885 kB] 2023-11-24T08:15:54.134 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.89.86 [2,336 B] 2023-11-24T08:15:54.216 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.89.86 [1,698 B] 2023-11-24T08:15:54.216 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.89.86 [2,492 B] 2023-11-24T08:15:54.216 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89 all 5.15.0-89.99 [12.3 MB] 2023-11-24T08:15:54.450 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-89-generic amd64 5.15.0-89.99 [2,885 kB] 2023-11-24T08:15:54.531 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.89.86 [2,336 B] 2023-11-24T08:15:54.549 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 113 MB in 3s (36.9 MB/s) 2023-11-24T08:15:54.716 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-5.15.0-89-generic. 2023-11-24T08:15:54.962 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 113 MB in 3s (34.4 MB/s) 2023-11-24T08:15:55.159 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-modules-5.15.0-89-generic. 2023-11-24T08:15:56.697 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-24T08:15:56.701 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../0-linux-modules-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-11-24T08:15:56.889 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:15:57.124 INFO:teuthology.orchestra.run.smithi164.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-24T08:15:57.128 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../0-linux-modules-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-11-24T08:15:57.197 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:15:59.325 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-image-5.15.0-89-generic. 2023-11-24T08:15:59.344 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../1-linux-image-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-11-24T08:15:59.450 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:15:59.498 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-image-5.15.0-89-generic. 2023-11-24T08:15:59.520 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../1-linux-image-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-11-24T08:15:59.611 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:15:59.909 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-89-generic. 2023-11-24T08:15:59.929 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-11-24T08:15:59.967 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:00.101 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-89-generic. 2023-11-24T08:16:00.122 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-11-24T08:16:00.159 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:06.628 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../3-linux-generic_5.15.0.89.86_amd64.deb ... 2023-11-24T08:16:06.736 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-11-24T08:16:06.794 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../3-linux-generic_5.15.0.89.86_amd64.deb ... 2023-11-24T08:16:06.936 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-11-24T08:16:07.062 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.89.86_amd64.deb ... 2023-11-24T08:16:07.183 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-11-24T08:16:07.251 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.89.86_amd64.deb ... 2023-11-24T08:16:07.381 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-image-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-11-24T08:16:07.503 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-89. 2023-11-24T08:16:07.525 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../5-linux-headers-5.15.0-89_5.15.0-89.99_all.deb ... 2023-11-24T08:16:07.561 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-11-24T08:16:07.667 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-headers-5.15.0-89. 2023-11-24T08:16:07.678 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../5-linux-headers-5.15.0-89_5.15.0-89.99_all.deb ... 2023-11-24T08:16:07.717 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-11-24T08:16:13.662 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-89-generic. 2023-11-24T08:16:13.700 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../6-linux-headers-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-11-24T08:16:13.736 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:13.755 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package linux-headers-5.15.0-89-generic. 2023-11-24T08:16:13.791 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../6-linux-headers-5.15.0-89-generic_5.15.0-89.99_amd64.deb ... 2023-11-24T08:16:13.829 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:16.425 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.89.86_amd64.deb ... 2023-11-24T08:16:16.525 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.89.86_amd64.deb ... 2023-11-24T08:16:16.537 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-11-24T08:16:16.648 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking linux-headers-generic (5.15.0.89.86) over (5.15.0.56.54) ... 2023-11-24T08:16:16.908 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-11-24T08:16:16.989 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-headers-5.15.0-89 (5.15.0-89.99) ... 2023-11-24T08:16:17.025 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:17.128 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-headers-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:17.151 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-generic (5.15.0.89.86) ... 2023-11-24T08:16:17.256 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:17.262 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-headers-generic (5.15.0.89.86) ... 2023-11-24T08:16:17.392 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:18.673 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-24T08:16:18.674 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-24T08:16:18.674 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-89-generic 2023-11-24T08:16:18.674 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-89-generic 2023-11-24T08:16:18.714 INFO:teuthology.orchestra.run.smithi164.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-24T08:16:18.714 INFO:teuthology.orchestra.run.smithi164.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-24T08:16:18.715 INFO:teuthology.orchestra.run.smithi164.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-89-generic 2023-11-24T08:16:18.715 INFO:teuthology.orchestra.run.smithi164.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-89-generic 2023-11-24T08:16:18.870 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:19.042 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-extra-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:20.279 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-image-generic (5.15.0.89.86) ... 2023-11-24T08:16:20.405 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-generic (5.15.0.89.86) ... 2023-11-24T08:16:20.476 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-generic (5.15.0.89.86) ... 2023-11-24T08:16:20.540 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:20.577 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-generic (5.15.0.89.86) ... 2023-11-24T08:16:20.678 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:21.950 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:22.007 INFO:teuthology.orchestra.run.smithi164.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-24T08:16:22.008 INFO:teuthology.orchestra.run.smithi164.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-89-generic 2023-11-24T08:16:22.072 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for linux-image-5.15.0-89-generic (5.15.0-89.99) ... 2023-11-24T08:16:22.138 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-24T08:16:22.138 INFO:teuthology.orchestra.run.smithi153.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-89-generic 2023-11-24T08:16:35.898 INFO:teuthology.orchestra.run.smithi164.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-24T08:16:35.899 INFO:teuthology.orchestra.run.smithi164.stdout:Sourcing file `/etc/default/grub' 2023-11-24T08:16:35.914 INFO:teuthology.orchestra.run.smithi164.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-24T08:16:35.978 INFO:teuthology.orchestra.run.smithi164.stdout:Generating grub configuration file ... 2023-11-24T08:16:36.195 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-24T08:16:36.196 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub' 2023-11-24T08:16:36.223 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-24T08:16:36.257 INFO:teuthology.orchestra.run.smithi153.stdout:Generating grub configuration file ... 2023-11-24T08:16:36.278 INFO:teuthology.orchestra.run.smithi164.stdout:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-11-24T08:16:36.299 INFO:teuthology.orchestra.run.smithi164.stdout:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-11-24T08:16:36.506 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-11-24T08:16:36.531 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-11-24T08:16:36.637 INFO:teuthology.orchestra.run.smithi164.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-24T08:16:36.643 INFO:teuthology.orchestra.run.smithi164.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-24T08:16:36.715 INFO:teuthology.orchestra.run.smithi164.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-24T08:16:36.721 INFO:teuthology.orchestra.run.smithi164.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-24T08:16:36.817 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-24T08:16:36.823 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-24T08:16:36.904 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-24T08:16:36.909 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-24T08:16:36.911 INFO:teuthology.orchestra.run.smithi164.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-24T08:16:36.912 INFO:teuthology.orchestra.run.smithi164.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-24T08:16:36.912 INFO:teuthology.orchestra.run.smithi164.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-24T08:16:36.954 INFO:teuthology.orchestra.run.smithi164.stdout:done 2023-11-24T08:16:37.074 INFO:teuthology.orchestra.run.smithi153.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-24T08:16:37.074 INFO:teuthology.orchestra.run.smithi153.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-24T08:16:37.074 INFO:teuthology.orchestra.run.smithi153.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-24T08:16:37.109 INFO:teuthology.orchestra.run.smithi153.stdout:done 2023-11-24T08:16:37.456 DEBUG:teuthology.orchestra.run.smithi164:> dpkg -s linux-image-generic 2023-11-24T08:16:37.479 INFO:teuthology.orchestra.run.smithi164.stdout:Package: linux-image-generic 2023-11-24T08:16:37.479 INFO:teuthology.orchestra.run.smithi164.stdout:Status: install ok installed 2023-11-24T08:16:37.479 INFO:teuthology.orchestra.run.smithi164.stdout:Priority: optional 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Section: kernel 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Installed-Size: 20 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Maintainer: Ubuntu Kernel Team 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: amd64 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Source: linux-meta 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Version: 5.15.0.89.86 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Recommends: thermald 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout:Description: Generic Linux kernel image 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout: This package will always depend on the latest generic kernel image 2023-11-24T08:16:37.480 INFO:teuthology.orchestra.run.smithi164.stdout: available. 2023-11-24T08:16:37.481 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-89-generic 2023-11-24T08:16:37.481 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-24T08:16:37.481 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-24T08:16:37.481 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-11-24T08:16:37.481 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2023-11-24T08:16:37.481 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get clean 2023-11-24T08:16:37.581 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2023-11-24T08:16:37.604 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2023-11-24T08:16:37.604 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2023-11-24T08:16:37.604 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2023-11-24T08:16:37.604 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 20 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.89.86 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2023-11-24T08:16:37.605 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2023-11-24T08:16:37.605 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-89-generic 2023-11-24T08:16:37.606 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-24T08:16:37.606 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-24T08:16:37.606 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-11-24T08:16:37.606 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2023-11-24T08:16:37.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2023-11-24T08:16:37.646 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get update 2023-11-24T08:16:37.791 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-24T08:16:37.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2023-11-24T08:16:37.912 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-24T08:16:37.921 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-24T08:16:37.922 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-24T08:16:37.936 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-24T08:16:37.956 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-24T08:16:37.996 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-24T08:16:38.074 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-24T08:16:39.135 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2023-11-24T08:16:39.152 DEBUG:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-24T08:16:39.167 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-24T08:16:39.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-24T08:16:39.210 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2023-11-24T08:16:39.242 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-24T08:16:39.391 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2023-11-24T08:16:39.391 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2023-11-24T08:16:39.435 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2023-11-24T08:16:39.435 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2023-11-24T08:16:39.535 INFO:teuthology.orchestra.run.smithi164.stdout:linux-image-generic is already the newest version (5.15.0.89.86). 2023-11-24T08:16:39.535 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages were automatically installed and are no longer required: 2023-11-24T08:16:39.536 INFO:teuthology.orchestra.run.smithi164.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-24T08:16:39.536 INFO:teuthology.orchestra.run.smithi164.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-24T08:16:39.536 INFO:teuthology.orchestra.run.smithi164.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-24T08:16:39.536 INFO:teuthology.orchestra.run.smithi164.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-24T08:16:39.536 INFO:teuthology.orchestra.run.smithi164.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-24T08:16:39.536 INFO:teuthology.orchestra.run.smithi164.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-24T08:16:39.573 INFO:teuthology.orchestra.run.smithi164.stdout:0 upgraded, 0 newly installed, 0 to remove and 273 not upgraded. 2023-11-24T08:16:39.575 DEBUG:teuthology.orchestra.run.smithi164:> dpkg -s linux-image-generic 2023-11-24T08:16:39.590 INFO:teuthology.orchestra.run.smithi153.stdout:linux-image-generic is already the newest version (5.15.0.89.86). 2023-11-24T08:16:39.590 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2023-11-24T08:16:39.590 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-24T08:16:39.590 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-24T08:16:39.590 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-24T08:16:39.590 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-24T08:16:39.590 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-24T08:16:39.590 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-24T08:16:39.626 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 0 newly installed, 0 to remove and 273 not upgraded. 2023-11-24T08:16:39.628 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Package: linux-image-generic 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Status: install ok installed 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Priority: optional 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Section: kernel 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Installed-Size: 20 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Maintainer: Ubuntu Kernel Team 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: amd64 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Source: linux-meta 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Version: 5.15.0.89.86 2023-11-24T08:16:39.632 INFO:teuthology.orchestra.run.smithi164.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-24T08:16:39.633 INFO:teuthology.orchestra.run.smithi164.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-24T08:16:39.633 INFO:teuthology.orchestra.run.smithi164.stdout:Recommends: thermald 2023-11-24T08:16:39.633 INFO:teuthology.orchestra.run.smithi164.stdout:Description: Generic Linux kernel image 2023-11-24T08:16:39.633 INFO:teuthology.orchestra.run.smithi164.stdout: This package will always depend on the latest generic kernel image 2023-11-24T08:16:39.633 INFO:teuthology.orchestra.run.smithi164.stdout: available. 2023-11-24T08:16:39.633 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2023-11-24T08:16:39.676 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.xveuECRAAm 2023-11-24T08:16:39.676 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub/grub.cfg /tmp/tmp.xveuECRAAm 2023-11-24T08:16:39.689 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2023-11-24T08:16:39.689 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2023-11-24T08:16:39.689 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2023-11-24T08:16:39.689 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2023-11-24T08:16:39.689 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 20 2023-11-24T08:16:39.689 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2023-11-24T08:16:39.689 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2023-11-24T08:16:39.690 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2023-11-24T08:16:39.690 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.89.86 2023-11-24T08:16:39.690 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-89), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-24T08:16:39.690 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-89-generic, linux-modules-extra-5.15.0-89-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-24T08:16:39.690 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2023-11-24T08:16:39.690 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2023-11-24T08:16:39.690 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2023-11-24T08:16:39.690 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2023-11-24T08:16:39.690 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-11-24T08:16:39.734 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.xveuECRAAm 2023-11-24T08:16:39.737 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.Qzn7hSyfRD 2023-11-24T08:16:39.738 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Qzn7hSyfRD 2023-11-24T08:16:39.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.Qzn7hSyfRD 2023-11-24T08:16:39.855 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.xveuECRAAm is 10KB 2023-11-24T08:16:39.867 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.xveuECRAAm 2023-11-24T08:16:39.872 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-24T08:16:39.930 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:16:39.931 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-24T08:16:39.932 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-24T08:16:39.934 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.Qzn7hSyfRD is 10KB 2023-11-24T08:16:39.990 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.Qzn7hSyfRD 2023-11-24T08:16:39.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-24T08:16:40.030 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-89-generic 2023-11-24T08:16:40.031 DEBUG:teuthology.orchestra.run.smithi164:> sudo update-grub 2023-11-24T08:16:40.049 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:16:40.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-24T08:16:40.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-24T08:16:40.117 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-89-generic 2023-11-24T08:16:40.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo update-grub 2023-11-24T08:16:41.201 INFO:teuthology.orchestra.run.smithi164.stderr:Sourcing file `/etc/default/grub' 2023-11-24T08:16:41.202 INFO:teuthology.orchestra.run.smithi164.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-24T08:16:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2023-11-24T08:16:41.302 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub' 2023-11-24T08:16:41.303 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-24T08:16:41.306 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2023-11-24T08:16:41.397 INFO:teuthology.orchestra.run.smithi164.stderr:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-11-24T08:16:41.405 INFO:teuthology.orchestra.run.smithi164.stderr:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-11-24T08:16:41.480 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-89-generic 2023-11-24T08:16:41.487 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-89-generic 2023-11-24T08:16:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-24T08:16:41.703 INFO:teuthology.orchestra.run.smithi164.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-24T08:16:41.740 INFO:teuthology.orchestra.run.smithi164.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-24T08:16:41.745 INFO:teuthology.orchestra.run.smithi164.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-24T08:16:41.762 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-24T08:16:41.767 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-24T08:16:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-24T08:16:41.815 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-24T08:16:41.900 INFO:teuthology.orchestra.run.smithi164.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-24T08:16:41.900 INFO:teuthology.orchestra.run.smithi164.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-24T08:16:41.900 INFO:teuthology.orchestra.run.smithi164.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-24T08:16:41.921 INFO:teuthology.orchestra.run.smithi164.stderr:done 2023-11-24T08:16:41.931 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-24T08:16:41.931 INFO:teuthology.orchestra.run.smithi153.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-24T08:16:41.932 INFO:teuthology.orchestra.run.smithi153.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-24T08:16:41.939 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2023-11-24T08:16:41.951 INFO:teuthology.orchestra.run.smithi153.stderr:done 2023-11-24T08:16:41.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2023-11-24T08:17:11.970 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-11-24T08:17:11.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-11-24T08:17:11.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:17:11.973 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-11-24T08:17:11.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-24T08:17:11.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:17:30.467 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-11-24T08:17:39.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-24T08:17:39.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:17:42.530 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-11-24T08:17:46.594 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-24T08:17:54.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-24T08:17:54.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:17:55.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-11-24T08:17:55.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:18:12.962 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-11-24T08:18:13.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-24T08:18:26.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-11-24T08:18:26.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:18:27.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-24T08:18:27.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:18:28.034 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-24T08:18:28.265 DEBUG:teuthology.orchestra.run.smithi153:> true 2023-11-24T08:18:29.392 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-24T08:18:31.543 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-89-generic"... 2023-11-24T08:18:31.544 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2023-11-24T08:18:31.551 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-89-generic 2023-11-24T08:18:31.551 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-89-generic vs 5.15.0-89-generic 2023-11-24T08:18:31.551 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-24T08:18:31.551 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-24T08:18:32.552 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-11-24T08:18:32.553 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-24T08:18:32.640 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2023-11-24T08:18:32.667 DEBUG:teuthology.parallel:result is None 2023-11-24T08:18:43.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-11-24T08:18:43.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:18:43.347 DEBUG:teuthology.orchestra.run.smithi164:> true 2023-11-24T08:18:43.924 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-11-24T08:18:43.924 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-89-generic"... 2023-11-24T08:18:43.924 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2023-11-24T08:18:43.972 INFO:teuthology.orchestra.run.smithi164.stdout:5.15.0-89-generic 2023-11-24T08:18:43.972 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-89-generic vs 5.15.0-89-generic 2023-11-24T08:18:43.972 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-24T08:18:43.972 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-24T08:18:44.973 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-11-24T08:18:44.973 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-24T08:18:45.063 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2023-11-24T08:18:45.094 DEBUG:teuthology.parallel:result is None 2023-11-24T08:18:45.094 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-24T08:18:45.107 INFO:teuthology.task.internal:Creating test directory... 2023-11-24T08:18:45.107 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-24T08:18:45.110 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-24T08:18:45.115 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-24T08:18:45.122 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-24T08:18:45.129 INFO:teuthology.task.internal:Creating archive directory... 2023-11-24T08:18:45.130 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-24T08:18:45.158 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-24T08:18:45.259 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-24T08:18:45.266 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-24T08:18:45.266 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-24T08:18:45.269 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-24T08:18:45.285 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T08:18:45.292 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T08:18:45.293 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T08:18:45.301 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T08:18:45.303 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-24T08:18:45.309 INFO:teuthology.task.internal:Configuring sudo... 2023-11-24T08:18:45.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-24T08:18:45.338 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-24T08:18:45.356 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-24T08:18:45.365 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-24T08:18:45.366 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-24T08:18:45.394 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-24T08:18:45.404 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T08:18:45.445 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T08:18:45.493 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:18:45.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-24T08:18:45.560 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T08:18:45.565 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T08:18:45.612 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:18:45.613 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-24T08:18:45.677 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2023-11-24T08:18:45.679 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2023-11-24T08:18:45.791 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-24T08:18:45.799 INFO:teuthology.task.internal:Starting timer... 2023-11-24T08:18:45.799 INFO:teuthology.run_tasks:Running task pcp... 2023-11-24T08:18:45.857 INFO:teuthology.run_tasks:Running task selinux... 2023-11-24T08:18:45.865 DEBUG:teuthology.task.selinux:Excluding smithi153: OS 'ubuntu' does not support SELinux 2023-11-24T08:18:45.865 DEBUG:teuthology.task.selinux:Excluding smithi164: OS 'ubuntu' does not support SELinux 2023-11-24T08:18:45.865 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-24T08:18:45.865 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-11-24T08:18:45.866 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-24T08:18:45.866 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-24T08:18:45.874 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-24T08:18:45.874 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-24T08:18:45.891 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-24T08:18:45.892 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-24T08:26:33.638 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2023-11-24T08:26:33.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-24T08:26:33.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:26:33.717 DEBUG:teuthology.orchestra.run.smithi153:> true 2023-11-24T08:26:33.822 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-24T08:26:33.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-11-24T08:26:33.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T08:26:33.898 DEBUG:teuthology.orchestra.run.smithi164:> true 2023-11-24T08:26:33.994 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-11-24T08:26:33.995 INFO:teuthology.run_tasks:Running task clock... 2023-11-24T08:26:34.007 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-24T08:26:34.007 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-24T08:26:34.008 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T08:26:34.011 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-24T08:26:34.011 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T08:26:34.038 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-24T08:26:34.039 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: Command line: ntpd -gq 2023-11-24T08:26:34.039 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: ---------------------------------------------------- 2023-11-24T08:26:34.039 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: ntp-4 is maintained by Network Time Foundation, 2023-11-24T08:26:34.039 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-24T08:26:34.039 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: corporation. Support and training for ntp-4 are 2023-11-24T08:26:34.039 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: available at https://www.nwtime.org/support 2023-11-24T08:26:34.039 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: ---------------------------------------------------- 2023-11-24T08:26:34.040 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: proto: precision = 0.045 usec (-24) 2023-11-24T08:26:34.040 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: basedate set to 2022-02-04 2023-11-24T08:26:34.040 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: gps base set to 2022-02-06 (week 2196) 2023-11-24T08:26:34.040 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-24T08:26:34.040 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: restrict ::: KOD does nothing without LIMITED. 2023-11-24T08:26:34.041 INFO:teuthology.orchestra.run.smithi153.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-24T08:26:34.041 INFO:teuthology.orchestra.run.smithi153.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-24T08:26:34.042 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: Listen and drop on 0 v6wildcard [::]:123 2023-11-24T08:26:34.042 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-24T08:26:34.042 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: Listen normally on 2 lo 127.0.0.1:123 2023-11-24T08:26:34.042 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: Listen normally on 3 enp3s0f1 172.21.15.153:123 2023-11-24T08:26:34.043 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: Listen normally on 4 lo [::1]:123 2023-11-24T08:26:34.043 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6deb%5]:123 2023-11-24T08:26:34.043 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:34 ntpd[27978]: Listening on routing socket on fd #22 for interface updates 2023-11-24T08:26:34.064 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-24T08:26:34.064 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: Command line: ntpd -gq 2023-11-24T08:26:34.064 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: ---------------------------------------------------- 2023-11-24T08:26:34.064 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: ntp-4 is maintained by Network Time Foundation, 2023-11-24T08:26:34.065 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-24T08:26:34.065 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: corporation. Support and training for ntp-4 are 2023-11-24T08:26:34.065 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: available at https://www.nwtime.org/support 2023-11-24T08:26:34.065 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: ---------------------------------------------------- 2023-11-24T08:26:34.065 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: proto: precision = 0.060 usec (-24) 2023-11-24T08:26:34.065 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: basedate set to 2022-02-04 2023-11-24T08:26:34.066 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: gps base set to 2022-02-06 (week 2196) 2023-11-24T08:26:34.066 INFO:teuthology.orchestra.run.smithi164.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-24T08:26:34.066 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-24T08:26:34.066 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: restrict ::: KOD does nothing without LIMITED. 2023-11-24T08:26:34.067 INFO:teuthology.orchestra.run.smithi164.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-24T08:26:34.067 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: Listen and drop on 0 v6wildcard [::]:123 2023-11-24T08:26:34.067 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-24T08:26:34.068 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: Listen normally on 2 lo 127.0.0.1:123 2023-11-24T08:26:34.068 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: Listen normally on 3 enp3s0f1 172.21.15.164:123 2023-11-24T08:26:34.068 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: Listen normally on 4 lo [::1]:123 2023-11-24T08:26:34.068 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:73f3%5]:123 2023-11-24T08:26:34.068 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:34 ntpd[28014]: Listening on routing socket on fd #22 for interface updates 2023-11-24T08:26:35.039 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:35.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:35.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:35 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:35.064 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:35 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:35.064 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:35 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:37.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:37.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:37.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:37.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:37.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:37 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:37 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:37 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:37 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:37 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:37 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:37 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:38.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:38.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:38.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:38.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:38.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:38.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:38.067 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:38.067 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:38.067 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:38.067 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:38.067 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:38.067 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:38 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:39.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:39.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:39.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:39.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:39.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:39.042 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:39 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:39.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:39 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:39.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:39 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:39.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:39 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:39.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:39 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:39.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:39 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:39.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:39 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:40.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:40.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:40.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:40.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:40.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:40.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:40.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:40.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:40.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:40.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:40.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:40.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:40 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:41.040 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:41.041 INFO:teuthology.orchestra.run.smithi153.stdout:24 Nov 08:26:41 ntpd[27978]: ntpd: time slew +0.000883 s 2023-11-24T08:26:41.041 INFO:teuthology.orchestra.run.smithi153.stdout:ntpd: time slew +0.000883s 2023-11-24T08:26:41.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-24T08:26:41.041 INFO:teuthology.orchestra.run.smithi153.stderr:24 Nov 08:26:41 ntpd[27978]: can't open /var/log/ntpstats/loopstats.20231124: Permission denied 2023-11-24T08:26:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:41 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:41 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:41 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:41 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:41 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:41 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:41.134 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2023-11-24T08:26:41.135 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2023-11-24T08:26:41.135 INFO:teuthology.orchestra.run.smithi153.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-24T08:26:41.135 INFO:teuthology.orchestra.run.smithi153.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-24T08:26:41.135 INFO:teuthology.orchestra.run.smithi153.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-24T08:26:41.135 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-24T08:26:42.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:42.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:42.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:42.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:42.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:42.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:42.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:42.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:42.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:42.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:42.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:42.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:42 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:43 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:43 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:43 ntpd[28014]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-24T08:26:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:43 ntpd[28014]: can't open /var/log/ntpstats/rawstats.20231124: Permission denied 2023-11-24T08:26:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:43 ntpd[28014]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-24T08:26:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:43 ntpd[28014]: can't open /var/log/ntpstats/peerstats.20231124: Permission denied 2023-11-24T08:26:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:24 Nov 08:26:43 ntpd[28014]: ntpd: time slew +0.002104 s 2023-11-24T08:26:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:ntpd: time slew +0.002104s 2023-11-24T08:26:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:43 ntpd[28014]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-24T08:26:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:24 Nov 08:26:43 ntpd[28014]: can't open /var/log/ntpstats/loopstats.20231124: Permission denied 2023-11-24T08:26:43.133 INFO:teuthology.orchestra.run.smithi164.stdout: remote refid st t when poll reach delay offset jitter 2023-11-24T08:26:43.133 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================== 2023-11-24T08:26:43.133 INFO:teuthology.orchestra.run.smithi164.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-24T08:26:43.133 INFO:teuthology.orchestra.run.smithi164.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-24T08:26:43.134 INFO:teuthology.orchestra.run.smithi164.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-24T08:26:43.134 INFO:teuthology.orchestra.run.smithi164.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-24T08:26:43.134 INFO:teuthology.run_tasks:Running task install... 2023-11-24T08:26:43.143 DEBUG:teuthology.task.install:project ceph 2023-11-24T08:26:43.143 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}} 2023-11-24T08:26:43.143 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'} 2023-11-24T08:26:43.144 INFO:teuthology.task.install:Using flavor: default 2023-11-24T08:26:43.149 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-24T08:26:43.149 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-11-24T08:26:43.150 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-key list | grep Ceph 2023-11-24T08:26:43.150 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-key list | grep Ceph 2023-11-24T08:26:43.214 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-24T08:26:43.231 INFO:teuthology.orchestra.run.smithi164.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-24T08:26:43.243 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-24T08:26:43.244 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph.com (release key) 2023-11-24T08:26:43.245 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-11-24T08:26:43.245 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 2023-11-24T08:26:43.259 INFO:teuthology.orchestra.run.smithi164.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-24T08:26:43.259 INFO:teuthology.orchestra.run.smithi164.stdout:uid [ unknown] Ceph.com (release key) 2023-11-24T08:26:43.260 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-11-24T08:26:43.260 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 2023-11-24T08:26:43.398 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default/ 2023-11-24T08:26:43.398 INFO:teuthology.task.install.deb:Package version is 18.0.0-7484-g135a1ac9-1jammy 2023-11-24T08:26:43.410 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default/ 2023-11-24T08:26:43.411 INFO:teuthology.task.install.deb:Package version is 18.0.0-7484-g135a1ac9-1jammy 2023-11-24T08:26:43.511 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:26:43.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-24T08:26:43.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2023-11-24T08:26:43.575 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:26:43.575 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-24T08:26:43.589 DEBUG:teuthology.orchestra.run.smithi164:> sudo apt-get update 2023-11-24T08:26:43.686 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-24T08:26:43.694 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-24T08:26:43.703 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-24T08:26:43.711 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-24T08:26:43.741 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy InRelease 2023-11-24T08:26:43.760 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-24T08:26:43.760 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-24T08:26:43.763 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2023-11-24T08:26:43.773 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-24T08:26:43.785 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-24T08:26:43.793 INFO:teuthology.orchestra.run.smithi164.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-24T08:26:43.806 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-11-24T08:26:43.833 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,406 B] 2023-11-24T08:26:43.869 INFO:teuthology.orchestra.run.smithi164.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy InRelease 2023-11-24T08:26:43.906 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2023-11-24T08:26:43.943 INFO:teuthology.orchestra.run.smithi164.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-24T08:26:43.979 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,406 B] 2023-11-24T08:26:44.021 INFO:teuthology.orchestra.run.smithi164.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-11-24T08:26:44.044 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 27.8 kB in 0s (80.3 kB/s) 2023-11-24T08:26:44.101 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 27.8 kB in 0s (82.3 kB/s) 2023-11-24T08:26:44.911 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-24T08:26:44.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7484-g135a1ac9-1jammy cephadm=18.0.0-7484-g135a1ac9-1jammy ceph-mds=18.0.0-7484-g135a1ac9-1jammy ceph-mgr=18.0.0-7484-g135a1ac9-1jammy ceph-common=18.0.0-7484-g135a1ac9-1jammy ceph-fuse=18.0.0-7484-g135a1ac9-1jammy ceph-test=18.0.0-7484-g135a1ac9-1jammy ceph-volume=18.0.0-7484-g135a1ac9-1jammy radosgw=18.0.0-7484-g135a1ac9-1jammy python3-rados=18.0.0-7484-g135a1ac9-1jammy python3-rgw=18.0.0-7484-g135a1ac9-1jammy python3-cephfs=18.0.0-7484-g135a1ac9-1jammy python3-rbd=18.0.0-7484-g135a1ac9-1jammy libcephfs2=18.0.0-7484-g135a1ac9-1jammy libcephfs-dev=18.0.0-7484-g135a1ac9-1jammy librados2=18.0.0-7484-g135a1ac9-1jammy librbd1=18.0.0-7484-g135a1ac9-1jammy rbd-fuse=18.0.0-7484-g135a1ac9-1jammy rbd-mirror=18.0.0-7484-g135a1ac9-1jammy 2023-11-24T08:26:44.985 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-24T08:26:45.005 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2023-11-24T08:26:45.020 DEBUG:teuthology.orchestra.run.smithi164:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7484-g135a1ac9-1jammy cephadm=18.0.0-7484-g135a1ac9-1jammy ceph-mds=18.0.0-7484-g135a1ac9-1jammy ceph-mgr=18.0.0-7484-g135a1ac9-1jammy ceph-common=18.0.0-7484-g135a1ac9-1jammy ceph-fuse=18.0.0-7484-g135a1ac9-1jammy ceph-test=18.0.0-7484-g135a1ac9-1jammy ceph-volume=18.0.0-7484-g135a1ac9-1jammy radosgw=18.0.0-7484-g135a1ac9-1jammy python3-rados=18.0.0-7484-g135a1ac9-1jammy python3-rgw=18.0.0-7484-g135a1ac9-1jammy python3-cephfs=18.0.0-7484-g135a1ac9-1jammy python3-rbd=18.0.0-7484-g135a1ac9-1jammy libcephfs2=18.0.0-7484-g135a1ac9-1jammy libcephfs-dev=18.0.0-7484-g135a1ac9-1jammy librados2=18.0.0-7484-g135a1ac9-1jammy librbd1=18.0.0-7484-g135a1ac9-1jammy rbd-fuse=18.0.0-7484-g135a1ac9-1jammy rbd-mirror=18.0.0-7484-g135a1ac9-1jammy 2023-11-24T08:26:45.079 INFO:teuthology.orchestra.run.smithi164.stdout:Reading package lists... 2023-11-24T08:26:45.173 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2023-11-24T08:26:45.173 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2023-11-24T08:26:45.264 INFO:teuthology.orchestra.run.smithi164.stdout:Building dependency tree... 2023-11-24T08:26:45.265 INFO:teuthology.orchestra.run.smithi164.stdout:Reading state information... 2023-11-24T08:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2023-11-24T08:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-24T08:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-24T08:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-24T08:26:45.335 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-24T08:26:45.337 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-24T08:26:45.337 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-24T08:26:45.337 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2023-11-24T08:26:45.337 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-24T08:26:45.337 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-24T08:26:45.337 INFO:teuthology.orchestra.run.smithi153.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liboath0 libonig5 libpcre2-16-0 2023-11-24T08:26:45.337 INFO:teuthology.orchestra.run.smithi153.stdout: libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 librdkafka1 librgw2 2023-11-24T08:26:45.337 INFO:teuthology.orchestra.run.smithi153.stdout: libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli python-asyncssh-doc 2023-11-24T08:26:45.337 INFO:teuthology.orchestra.run.smithi153.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common python3-certifi python3-cheroot python3-cherrypy3 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth python3-idna python3-iniconfig python3-jaraco.classes 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytest python3-repoze.lru python3-requests python3-requests-oauthlib 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2023-11-24T08:26:45.338 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2023-11-24T08:26:45.339 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2023-11-24T08:26:45.339 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2023-11-24T08:26:45.339 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2023-11-24T08:26:45.339 INFO:teuthology.orchestra.run.smithi153.stdout: socat xmlstarlet 2023-11-24T08:26:45.339 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2023-11-24T08:26:45.339 INFO:teuthology.orchestra.run.smithi153.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-24T08:26:45.339 INFO:teuthology.orchestra.run.smithi153.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-24T08:26:45.339 INFO:teuthology.orchestra.run.smithi153.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-24T08:26:45.340 INFO:teuthology.orchestra.run.smithi153.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-24T08:26:45.340 INFO:teuthology.orchestra.run.smithi153.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-24T08:26:45.340 INFO:teuthology.orchestra.run.smithi153.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-24T08:26:45.340 INFO:teuthology.orchestra.run.smithi153.stdout: python3-watchdog 2023-11-24T08:26:45.340 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2023-11-24T08:26:45.340 INFO:teuthology.orchestra.run.smithi153.stdout: btrfs-tools 2023-11-24T08:26:45.435 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2023-11-24T08:26:45.436 INFO:teuthology.orchestra.run.smithi153.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-24T08:26:45.436 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-24T08:26:45.436 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-24T08:26:45.438 INFO:teuthology.orchestra.run.smithi164.stdout:The following packages were automatically installed and are no longer required: 2023-11-24T08:26:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-24T08:26:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-24T08:26:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-24T08:26:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-24T08:26:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-24T08:26:45.438 INFO:teuthology.orchestra.run.smithi164.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-24T08:26:45.438 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 liboath0 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-24T08:26:45.439 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-24T08:26:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-24T08:26:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-24T08:26:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-24T08:26:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-24T08:26:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-24T08:26:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-24T08:26:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-24T08:26:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-24T08:26:45.440 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror socat xmlstarlet 2023-11-24T08:26:45.443 INFO:teuthology.orchestra.run.smithi164.stdout:The following additional packages will be installed: 2023-11-24T08:26:45.443 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-24T08:26:45.443 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liboath0 libonig5 libpcre2-16-0 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 librdkafka1 librgw2 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli python-asyncssh-doc 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common python3-certifi python3-cheroot python3-cherrypy3 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth python3-idna python3-iniconfig python3-jaraco.classes 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-24T08:26:45.444 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytest python3-repoze.lru python3-requests python3-requests-oauthlib 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: socat xmlstarlet 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout:Suggested packages: 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-24T08:26:45.445 INFO:teuthology.orchestra.run.smithi164.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-24T08:26:45.446 INFO:teuthology.orchestra.run.smithi164.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-24T08:26:45.446 INFO:teuthology.orchestra.run.smithi164.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-24T08:26:45.446 INFO:teuthology.orchestra.run.smithi164.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-24T08:26:45.446 INFO:teuthology.orchestra.run.smithi164.stdout: python3-watchdog 2023-11-24T08:26:45.446 INFO:teuthology.orchestra.run.smithi164.stdout:Recommended packages: 2023-11-24T08:26:45.446 INFO:teuthology.orchestra.run.smithi164.stdout: btrfs-tools 2023-11-24T08:26:45.498 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 109 newly installed, 0 to remove and 261 not upgraded. 2023-11-24T08:26:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 188 MB of archives. 2023-11-24T08:26:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 840 MB of additional disk space will be used. 2023-11-24T08:26:45.499 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-24T08:26:45.541 INFO:teuthology.orchestra.run.smithi164.stdout:The following NEW packages will be installed: 2023-11-24T08:26:45.541 INFO:teuthology.orchestra.run.smithi164.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-24T08:26:45.541 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-24T08:26:45.541 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-24T08:26:45.543 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 liboath0 2023-11-24T08:26:45.543 INFO:teuthology.orchestra.run.smithi164.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-11-24T08:26:45.543 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-24T08:26:45.543 INFO:teuthology.orchestra.run.smithi164.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-24T08:26:45.544 INFO:teuthology.orchestra.run.smithi164.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-24T08:26:45.545 INFO:teuthology.orchestra.run.smithi164.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-24T08:26:45.545 INFO:teuthology.orchestra.run.smithi164.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-24T08:26:45.545 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-24T08:26:45.545 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-24T08:26:45.545 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror socat xmlstarlet 2023-11-24T08:26:45.601 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-24T08:26:45.603 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-24T08:26:45.623 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-24T08:26:45.657 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7484-g135a1ac9-1jammy [3,458 kB] 2023-11-24T08:26:45.685 INFO:teuthology.orchestra.run.smithi164.stdout:0 upgraded, 109 newly installed, 0 to remove and 261 not upgraded. 2023-11-24T08:26:45.685 INFO:teuthology.orchestra.run.smithi164.stdout:Need to get 188 MB of archives. 2023-11-24T08:26:45.686 INFO:teuthology.orchestra.run.smithi164.stdout:After this operation, 840 MB of additional disk space will be used. 2023-11-24T08:26:45.686 INFO:teuthology.orchestra.run.smithi164.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7484-g135a1ac9-1jammy [3,458 kB] 2023-11-24T08:26:45.692 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-24T08:26:45.697 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-24T08:26:45.710 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-24T08:26:45.714 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-24T08:26:45.714 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-24T08:26:45.715 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-24T08:26:45.723 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-24T08:26:45.724 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-24T08:26:45.724 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-24T08:26:45.725 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-24T08:26:45.730 INFO:teuthology.orchestra.run.smithi164.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-24T08:26:45.746 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-24T08:26:45.746 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-24T08:26:45.747 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-24T08:26:45.747 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-24T08:26:45.748 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-24T08:26:45.748 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-24T08:26:45.751 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-24T08:26:45.765 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-24T08:26:45.766 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-24T08:26:45.766 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-24T08:26:45.785 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-24T08:26:45.786 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-24T08:26:45.786 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-24T08:26:45.787 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-24T08:26:45.788 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-24T08:26:45.789 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-24T08:26:45.795 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-24T08:26:45.806 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-24T08:26:45.806 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-24T08:26:45.807 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-24T08:26:45.825 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-24T08:26:45.828 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-24T08:26:45.829 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-11-24T08:26:45.830 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-24T08:26:45.831 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-24T08:26:45.834 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-24T08:26:45.834 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-24T08:26:45.846 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-24T08:26:45.913 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-24T08:26:45.917 INFO:teuthology.orchestra.run.smithi153.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-24T08:26:45.919 INFO:teuthology.orchestra.run.smithi153.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-24T08:26:45.919 INFO:teuthology.orchestra.run.smithi153.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-24T08:26:45.920 INFO:teuthology.orchestra.run.smithi153.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-24T08:26:45.921 INFO:teuthology.orchestra.run.smithi153.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-24T08:26:45.928 INFO:teuthology.orchestra.run.smithi164.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7484-g135a1ac9-1jammy [3,112 kB] 2023-11-24T08:26:45.953 INFO:teuthology.orchestra.run.smithi153.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-24T08:26:45.955 INFO:teuthology.orchestra.run.smithi153.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-24T08:26:45.956 INFO:teuthology.orchestra.run.smithi153.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-24T08:26:45.984 INFO:teuthology.orchestra.run.smithi153.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-24T08:26:45.994 INFO:teuthology.orchestra.run.smithi153.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-24T08:26:45.995 INFO:teuthology.orchestra.run.smithi153.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-24T08:26:45.996 INFO:teuthology.orchestra.run.smithi153.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-24T08:26:45.997 INFO:teuthology.orchestra.run.smithi153.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-24T08:26:45.997 INFO:teuthology.orchestra.run.smithi153.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-24T08:26:45.998 INFO:teuthology.orchestra.run.smithi153.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-24T08:26:45.998 INFO:teuthology.orchestra.run.smithi153.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-24T08:26:46.003 INFO:teuthology.orchestra.run.smithi153.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-24T08:26:46.027 INFO:teuthology.orchestra.run.smithi153.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-24T08:26:46.029 INFO:teuthology.orchestra.run.smithi153.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-24T08:26:46.030 INFO:teuthology.orchestra.run.smithi153.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-24T08:26:46.034 INFO:teuthology.orchestra.run.smithi153.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-24T08:26:46.037 INFO:teuthology.orchestra.run.smithi153.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-24T08:26:46.043 INFO:teuthology.orchestra.run.smithi153.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-24T08:26:46.046 INFO:teuthology.orchestra.run.smithi153.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-24T08:26:46.047 INFO:teuthology.orchestra.run.smithi153.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-24T08:26:46.057 INFO:teuthology.orchestra.run.smithi153.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-24T08:26:46.057 INFO:teuthology.orchestra.run.smithi153.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-24T08:26:46.062 INFO:teuthology.orchestra.run.smithi164.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7484-g135a1ac9-1jammy [691 kB] 2023-11-24T08:26:46.077 INFO:teuthology.orchestra.run.smithi153.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-24T08:26:46.078 INFO:teuthology.orchestra.run.smithi153.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-24T08:26:46.086 INFO:teuthology.orchestra.run.smithi153.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-24T08:26:46.087 INFO:teuthology.orchestra.run.smithi153.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-24T08:26:46.088 INFO:teuthology.orchestra.run.smithi153.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-24T08:26:46.090 INFO:teuthology.orchestra.run.smithi164.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7484-g135a1ac9-1jammy [357 kB] 2023-11-24T08:26:46.092 INFO:teuthology.orchestra.run.smithi153.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-24T08:26:46.092 INFO:teuthology.orchestra.run.smithi153.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-24T08:26:46.104 INFO:teuthology.orchestra.run.smithi164.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7484-g135a1ac9-1jammy [32.8 kB] 2023-11-24T08:26:46.106 INFO:teuthology.orchestra.run.smithi164.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7484-g135a1ac9-1jammy [180 kB] 2023-11-24T08:26:46.111 INFO:teuthology.orchestra.run.smithi164.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7484-g135a1ac9-1jammy [63.5 kB] 2023-11-24T08:26:46.113 INFO:teuthology.orchestra.run.smithi164.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7484-g135a1ac9-1jammy [330 kB] 2023-11-24T08:26:46.124 INFO:teuthology.orchestra.run.smithi164.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7484-g135a1ac9-1jammy [6,230 kB] 2023-11-24T08:26:46.158 INFO:teuthology.orchestra.run.smithi164.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-24T08:26:46.169 INFO:teuthology.orchestra.run.smithi153.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7484-g135a1ac9-1jammy [3,112 kB] 2023-11-24T08:26:46.170 INFO:teuthology.orchestra.run.smithi164.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-24T08:26:46.252 INFO:teuthology.orchestra.run.smithi164.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-24T08:26:46.433 INFO:teuthology.orchestra.run.smithi153.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7484-g135a1ac9-1jammy [691 kB] 2023-11-24T08:26:46.442 INFO:teuthology.orchestra.run.smithi164.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7484-g135a1ac9-1jammy [112 kB] 2023-11-24T08:26:46.447 INFO:teuthology.orchestra.run.smithi164.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7484-g135a1ac9-1jammy [439 kB] 2023-11-24T08:26:46.469 INFO:teuthology.orchestra.run.smithi164.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7484-g135a1ac9-1jammy [22.6 MB] 2023-11-24T08:26:46.482 INFO:teuthology.orchestra.run.smithi164.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-24T08:26:46.493 INFO:teuthology.orchestra.run.smithi153.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7484-g135a1ac9-1jammy [357 kB] 2023-11-24T08:26:46.494 INFO:teuthology.orchestra.run.smithi164.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-24T08:26:46.523 INFO:teuthology.orchestra.run.smithi153.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7484-g135a1ac9-1jammy [32.8 kB] 2023-11-24T08:26:46.524 INFO:teuthology.orchestra.run.smithi164.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-24T08:26:46.527 INFO:teuthology.orchestra.run.smithi153.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7484-g135a1ac9-1jammy [180 kB] 2023-11-24T08:26:46.533 INFO:teuthology.orchestra.run.smithi164.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-24T08:26:46.534 INFO:teuthology.orchestra.run.smithi164.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-24T08:26:46.535 INFO:teuthology.orchestra.run.smithi164.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-24T08:26:46.542 INFO:teuthology.orchestra.run.smithi153.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7484-g135a1ac9-1jammy [63.5 kB] 2023-11-24T08:26:46.548 INFO:teuthology.orchestra.run.smithi153.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7484-g135a1ac9-1jammy [330 kB] 2023-11-24T08:26:46.554 INFO:teuthology.orchestra.run.smithi164.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-24T08:26:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7484-g135a1ac9-1jammy [6,230 kB] 2023-11-24T08:26:46.719 INFO:teuthology.orchestra.run.smithi164.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-24T08:26:46.802 INFO:teuthology.orchestra.run.smithi164.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-24T08:26:46.986 INFO:teuthology.orchestra.run.smithi164.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-24T08:26:46.995 INFO:teuthology.orchestra.run.smithi164.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-24T08:26:47.004 INFO:teuthology.orchestra.run.smithi164.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-24T08:26:47.018 INFO:teuthology.orchestra.run.smithi164.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-24T08:26:47.036 INFO:teuthology.orchestra.run.smithi164.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-24T08:26:47.044 INFO:teuthology.orchestra.run.smithi164.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-24T08:26:47.053 INFO:teuthology.orchestra.run.smithi164.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-24T08:26:47.156 INFO:teuthology.orchestra.run.smithi153.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7484-g135a1ac9-1jammy [112 kB] 2023-11-24T08:26:47.165 INFO:teuthology.orchestra.run.smithi164.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-24T08:26:47.166 INFO:teuthology.orchestra.run.smithi153.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7484-g135a1ac9-1jammy [439 kB] 2023-11-24T08:26:47.177 INFO:teuthology.orchestra.run.smithi164.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-24T08:26:47.207 INFO:teuthology.orchestra.run.smithi153.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7484-g135a1ac9-1jammy [22.6 MB] 2023-11-24T08:26:47.347 INFO:teuthology.orchestra.run.smithi164.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-24T08:26:47.430 INFO:teuthology.orchestra.run.smithi164.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-24T08:26:47.573 INFO:teuthology.orchestra.run.smithi164.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-24T08:26:47.594 INFO:teuthology.orchestra.run.smithi164.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-24T08:26:47.603 INFO:teuthology.orchestra.run.smithi164.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-24T08:26:47.680 INFO:teuthology.orchestra.run.smithi164.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7484-g135a1ac9-1jammy [6,900 kB] 2023-11-24T08:26:47.683 INFO:teuthology.orchestra.run.smithi164.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-24T08:26:47.713 INFO:teuthology.orchestra.run.smithi164.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-24T08:26:47.721 INFO:teuthology.orchestra.run.smithi164.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-24T08:26:47.849 INFO:teuthology.orchestra.run.smithi164.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-24T08:26:47.850 INFO:teuthology.orchestra.run.smithi164.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-24T08:26:47.854 INFO:teuthology.orchestra.run.smithi164.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-24T08:26:47.947 INFO:teuthology.orchestra.run.smithi164.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7484-g135a1ac9-1jammy [238 kB] 2023-11-24T08:26:47.958 INFO:teuthology.orchestra.run.smithi164.stdout:Get:48 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7484-g135a1ac9-1jammy [124 kB] 2023-11-24T08:26:47.965 INFO:teuthology.orchestra.run.smithi164.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7484-g135a1ac9-1jammy [1,495 kB] 2023-11-24T08:26:48.024 INFO:teuthology.orchestra.run.smithi164.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-24T08:26:48.035 INFO:teuthology.orchestra.run.smithi164.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7484-g135a1ac9-1jammy [9,955 kB] 2023-11-24T08:26:48.346 INFO:teuthology.orchestra.run.smithi164.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-24T08:26:48.464 INFO:teuthology.orchestra.run.smithi164.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-24T08:26:48.465 INFO:teuthology.orchestra.run.smithi164.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7484-g135a1ac9-1jammy [31.8 MB] 2023-11-24T08:26:48.498 INFO:teuthology.orchestra.run.smithi164.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-11-24T08:26:48.532 INFO:teuthology.orchestra.run.smithi164.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-24T08:26:48.546 INFO:teuthology.orchestra.run.smithi164.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-24T08:26:48.599 INFO:teuthology.orchestra.run.smithi164.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-24T08:26:48.608 INFO:teuthology.orchestra.run.smithi164.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-24T08:26:48.638 INFO:teuthology.orchestra.run.smithi164.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-24T08:26:48.872 INFO:teuthology.orchestra.run.smithi164.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-24T08:26:48.873 INFO:teuthology.orchestra.run.smithi164.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-24T08:26:49.040 INFO:teuthology.orchestra.run.smithi164.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-24T08:26:49.236 INFO:teuthology.orchestra.run.smithi164.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-24T08:26:49.255 INFO:teuthology.orchestra.run.smithi164.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-24T08:26:49.404 INFO:teuthology.orchestra.run.smithi164.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-24T08:26:49.747 INFO:teuthology.orchestra.run.smithi164.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-24T08:26:49.757 INFO:teuthology.orchestra.run.smithi164.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-24T08:26:49.758 INFO:teuthology.orchestra.run.smithi164.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-24T08:26:49.825 INFO:teuthology.orchestra.run.smithi164.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-24T08:26:49.826 INFO:teuthology.orchestra.run.smithi164.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-24T08:26:49.828 INFO:teuthology.orchestra.run.smithi164.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-24T08:26:49.829 INFO:teuthology.orchestra.run.smithi164.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-24T08:26:49.980 INFO:teuthology.orchestra.run.smithi153.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7484-g135a1ac9-1jammy [6,900 kB] 2023-11-24T08:26:49.994 INFO:teuthology.orchestra.run.smithi164.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi164.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7484-g135a1ac9-1jammy [14.0 kB] 2023-11-24T08:26:50.029 INFO:teuthology.orchestra.run.smithi164.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7484-g135a1ac9-1jammy [839 kB] 2023-11-24T08:26:50.046 INFO:teuthology.orchestra.run.smithi164.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7484-g135a1ac9-1jammy [2,144 kB] 2023-11-24T08:26:50.084 INFO:teuthology.orchestra.run.smithi164.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7484-g135a1ac9-1jammy [802 kB] 2023-11-24T08:26:50.095 INFO:teuthology.orchestra.run.smithi164.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7484-g135a1ac9-1jammy [139 kB] 2023-11-24T08:26:50.097 INFO:teuthology.orchestra.run.smithi164.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7484-g135a1ac9-1jammy [1,900 kB] 2023-11-24T08:26:50.135 INFO:teuthology.orchestra.run.smithi164.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7484-g135a1ac9-1jammy [8,625 kB] 2023-11-24T08:26:50.195 INFO:teuthology.orchestra.run.smithi164.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-24T08:26:50.224 INFO:teuthology.orchestra.run.smithi164.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-24T08:26:50.288 INFO:teuthology.orchestra.run.smithi164.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-24T08:26:50.494 INFO:teuthology.orchestra.run.smithi164.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7484-g135a1ac9-1jammy [14.2 kB] 2023-11-24T08:26:50.494 INFO:teuthology.orchestra.run.smithi164.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7484-g135a1ac9-1jammy [48.6 MB] 2023-11-24T08:26:50.511 INFO:teuthology.orchestra.run.smithi164.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-24T08:26:50.550 INFO:teuthology.orchestra.run.smithi164.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-24T08:26:50.575 INFO:teuthology.orchestra.run.smithi164.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-24T08:26:50.584 INFO:teuthology.orchestra.run.smithi164.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-24T08:26:50.628 INFO:teuthology.orchestra.run.smithi164.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-24T08:26:50.654 INFO:teuthology.orchestra.run.smithi164.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-24T08:26:50.687 INFO:teuthology.orchestra.run.smithi164.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-24T08:26:50.869 INFO:teuthology.orchestra.run.smithi164.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-24T08:26:50.870 INFO:teuthology.orchestra.run.smithi153.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7484-g135a1ac9-1jammy [238 kB] 2023-11-24T08:26:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7484-g135a1ac9-1jammy [124 kB] 2023-11-24T08:26:50.926 INFO:teuthology.orchestra.run.smithi153.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7484-g135a1ac9-1jammy [1,495 kB] 2023-11-24T08:26:50.952 INFO:teuthology.orchestra.run.smithi164.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-24T08:26:51.124 INFO:teuthology.orchestra.run.smithi164.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-24T08:26:51.141 INFO:teuthology.orchestra.run.smithi153.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7484-g135a1ac9-1jammy [9,955 kB] 2023-11-24T08:26:51.152 INFO:teuthology.orchestra.run.smithi164.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-24T08:26:51.289 INFO:teuthology.orchestra.run.smithi164.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-24T08:26:51.322 INFO:teuthology.orchestra.run.smithi164.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-24T08:26:51.469 INFO:teuthology.orchestra.run.smithi164.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-24T08:26:51.473 INFO:teuthology.orchestra.run.smithi164.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-24T08:26:51.473 INFO:teuthology.orchestra.run.smithi164.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-24T08:26:51.494 INFO:teuthology.orchestra.run.smithi164.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-24T08:26:51.498 INFO:teuthology.orchestra.run.smithi164.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-24T08:26:52.489 INFO:teuthology.orchestra.run.smithi153.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7484-g135a1ac9-1jammy [31.8 MB] 2023-11-24T08:26:52.810 INFO:teuthology.orchestra.run.smithi164.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7484-g135a1ac9-1jammy [121 kB] 2023-11-24T08:26:52.812 INFO:teuthology.orchestra.run.smithi164.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7484-g135a1ac9-1jammy [38.3 kB] 2023-11-24T08:26:52.813 INFO:teuthology.orchestra.run.smithi164.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7484-g135a1ac9-1jammy [12.3 MB] 2023-11-24T08:26:53.214 INFO:teuthology.orchestra.run.smithi164.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7484-g135a1ac9-1jammy [91.4 kB] 2023-11-24T08:26:53.216 INFO:teuthology.orchestra.run.smithi164.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 18.0.0-7484-g135a1ac9-1jammy [3,049 kB] 2023-11-24T08:26:53.888 INFO:teuthology.orchestra.run.smithi164.stdout:Fetched 188 MB in 8s (24.2 MB/s) 2023-11-24T08:26:54.053 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-24T08:26:54.153 INFO:teuthology.orchestra.run.smithi164.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165983 files and directories currently installed.) 2023-11-24T08:26:54.158 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-24T08:26:54.244 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-24T08:26:54.581 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-24T08:26:54.607 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-24T08:26:54.647 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-24T08:26:54.975 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-24T08:26:54.981 INFO:teuthology.orchestra.run.smithi153.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7484-g135a1ac9-1jammy [14.0 kB] 2023-11-24T08:26:54.982 INFO:teuthology.orchestra.run.smithi153.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7484-g135a1ac9-1jammy [839 kB] 2023-11-24T08:26:54.998 INFO:teuthology.orchestra.run.smithi153.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7484-g135a1ac9-1jammy [2,144 kB] 2023-11-24T08:26:55.001 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-24T08:26:55.041 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-24T08:26:55.123 INFO:teuthology.orchestra.run.smithi153.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7484-g135a1ac9-1jammy [802 kB] 2023-11-24T08:26:55.165 INFO:teuthology.orchestra.run.smithi153.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7484-g135a1ac9-1jammy [139 kB] 2023-11-24T08:26:55.171 INFO:teuthology.orchestra.run.smithi153.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7484-g135a1ac9-1jammy [1,900 kB] 2023-11-24T08:26:55.287 INFO:teuthology.orchestra.run.smithi153.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7484-g135a1ac9-1jammy [8,625 kB] 2023-11-24T08:26:55.454 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-24T08:26:55.480 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-24T08:26:55.529 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:26:55.857 INFO:teuthology.orchestra.run.smithi153.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7484-g135a1ac9-1jammy [14.2 kB] 2023-11-24T08:26:55.858 INFO:teuthology.orchestra.run.smithi153.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7484-g135a1ac9-1jammy [48.6 MB] 2023-11-24T08:26:55.932 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-24T08:26:55.958 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-24T08:26:55.998 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:26:56.351 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-24T08:26:56.378 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-24T08:26:56.409 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:26:56.746 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-24T08:26:56.772 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-24T08:26:56.812 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-24T08:26:57.132 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librados2. 2023-11-24T08:26:57.158 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../007-librados2_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:26:57.199 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librados2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:26:57.610 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librbd1. 2023-11-24T08:26:57.625 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../008-librbd1_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:26:57.668 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librbd1 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:26:58.105 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libcephfs2. 2023-11-24T08:26:58.131 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../009-libcephfs2_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:26:58.171 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libcephfs2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:26:58.448 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rados. 2023-11-24T08:26:58.463 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../010-python3-rados_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:26:58.507 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rados (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:26:58.810 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-24T08:26:58.836 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:26:58.868 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-ceph-argparse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:26:59.070 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cephfs. 2023-11-24T08:26:59.085 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../012-python3-cephfs_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:26:59.129 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cephfs (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:26:59.440 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-24T08:26:59.466 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../013-python3-ceph-common_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:26:59.506 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-ceph-common (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:26:59.851 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-24T08:26:59.878 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-24T08:26:59.918 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-24T08:27:00.150 INFO:teuthology.orchestra.run.smithi153.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7484-g135a1ac9-1jammy [121 kB] 2023-11-24T08:27:00.153 INFO:teuthology.orchestra.run.smithi153.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7484-g135a1ac9-1jammy [38.3 kB] 2023-11-24T08:27:00.154 INFO:teuthology.orchestra.run.smithi153.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7484-g135a1ac9-1jammy [12.3 MB] 2023-11-24T08:27:00.204 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-prettytable. 2023-11-24T08:27:00.230 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2023-11-24T08:27:00.270 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-24T08:27:00.481 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rbd. 2023-11-24T08:27:00.496 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../016-python3-rbd_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:00.540 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rbd (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:00.860 INFO:teuthology.orchestra.run.smithi153.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7484-g135a1ac9-1jammy [91.4 kB] 2023-11-24T08:27:00.862 INFO:teuthology.orchestra.run.smithi153.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-dis-testing/135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 18.0.0-7484-g135a1ac9-1jammy [3,049 kB] 2023-11-24T08:27:00.884 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-24T08:27:00.910 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-24T08:27:00.951 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-24T08:27:01.262 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package librgw2. 2023-11-24T08:27:01.288 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../018-librgw2_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:01.329 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking librgw2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:01.570 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 188 MB in 16s (12.1 MB/s) 2023-11-24T08:27:01.741 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-24T08:27:01.840 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165983 files and directories currently installed.) 2023-11-24T08:27:01.846 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-24T08:27:01.906 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rgw. 2023-11-24T08:27:01.931 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-24T08:27:01.933 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../019-python3-rgw_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:01.964 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rgw (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:02.268 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-24T08:27:02.293 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-24T08:27:02.294 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-24T08:27:02.319 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../020-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-24T08:27:02.334 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-24T08:27:02.359 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-24T08:27:02.671 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libradosstriper1. 2023-11-24T08:27:02.696 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-24T08:27:02.697 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../021-libradosstriper1_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:02.722 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-24T08:27:02.737 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libradosstriper1 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:02.762 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-24T08:27:03.082 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-common. 2023-11-24T08:27:03.108 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../022-ceph-common_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:03.141 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-24T08:27:03.149 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-common (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:03.168 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-24T08:27:03.215 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:27:03.777 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-24T08:27:03.803 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-24T08:27:03.844 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:27:04.164 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-24T08:27:04.190 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-24T08:27:04.222 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:27:04.463 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-base. 2023-11-24T08:27:04.490 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../023-ceph-base_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:04.539 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-base (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:04.567 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-24T08:27:04.593 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-24T08:27:04.633 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-24T08:27:04.961 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librados2. 2023-11-24T08:27:04.987 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../007-librados2_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:05.027 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librados2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:05.200 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-24T08:27:05.227 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../024-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-24T08:27:05.267 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-24T08:27:05.472 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librbd1. 2023-11-24T08:27:05.498 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../008-librbd1_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:05.539 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librbd1 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:05.578 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cheroot. 2023-11-24T08:27:05.604 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../025-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-24T08:27:05.645 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-24T08:27:05.917 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs2. 2023-11-24T08:27:05.933 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../009-libcephfs2_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:05.964 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-24T08:27:05.966 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:05.991 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../026-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-24T08:27:06.031 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-24T08:27:06.244 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rados. 2023-11-24T08:27:06.267 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../010-python3-rados_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:06.267 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-24T08:27:06.282 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../027-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-24T08:27:06.310 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rados (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:06.317 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-24T08:27:06.605 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-24T08:27:06.628 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-24T08:27:06.631 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:06.655 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../028-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-24T08:27:06.671 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-argparse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:06.695 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-24T08:27:06.924 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cephfs. 2023-11-24T08:27:06.939 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../012-python3-cephfs_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:06.982 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cephfs (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:07.215 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-tempora. 2023-11-24T08:27:07.241 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../029-python3-tempora_4.1.2-1_all.deb ... 2023-11-24T08:27:07.281 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-24T08:27:07.285 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-24T08:27:07.311 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../013-python3-ceph-common_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:07.351 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-common (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:07.551 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-portend. 2023-11-24T08:27:07.577 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../030-python3-portend_3.0.0-1_all.deb ... 2023-11-24T08:27:07.609 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-24T08:27:07.680 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-24T08:27:07.706 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-24T08:27:07.746 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-24T08:27:07.903 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-24T08:27:07.930 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../031-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-24T08:27:07.970 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-24T08:27:08.033 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-prettytable. 2023-11-24T08:27:08.059 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2023-11-24T08:27:08.091 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-24T08:27:08.273 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-24T08:27:08.300 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../032-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-24T08:27:08.340 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-24T08:27:08.343 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rbd. 2023-11-24T08:27:08.369 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../016-python3-rbd_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:08.410 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rbd (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:08.652 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-natsort. 2023-11-24T08:27:08.678 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../033-python3-natsort_8.0.2-1_all.deb ... 2023-11-24T08:27:08.718 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-24T08:27:08.763 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-24T08:27:08.793 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-24T08:27:08.929 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-24T08:27:09.030 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-logutils. 2023-11-24T08:27:09.057 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../034-python3-logutils_0.3.3-8_all.deb ... 2023-11-24T08:27:09.096 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-24T08:27:09.282 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librgw2. 2023-11-24T08:27:09.309 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../018-librgw2_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:09.349 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librgw2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:09.399 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-24T08:27:09.426 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../035-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-24T08:27:09.466 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-24T08:27:09.744 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-mako. 2023-11-24T08:27:09.759 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../036-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-24T08:27:09.793 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-24T08:27:09.818 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rgw. 2023-11-24T08:27:09.833 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../019-python3-rgw_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:09.868 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rgw (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:10.122 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-24T08:27:10.148 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../037-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-24T08:27:10.188 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-24T08:27:10.204 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-24T08:27:10.230 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../020-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-24T08:27:10.271 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-24T08:27:10.474 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-24T08:27:10.501 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../038-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-24T08:27:10.541 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-24T08:27:10.565 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libradosstriper1. 2023-11-24T08:27:10.591 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../021-libradosstriper1_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:10.632 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libradosstriper1 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:10.819 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-webob. 2023-11-24T08:27:10.841 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../039-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-24T08:27:10.877 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-24T08:27:11.068 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-common. 2023-11-24T08:27:11.095 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../022-ceph-common_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:11.135 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-common (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:11.163 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-waitress. 2023-11-24T08:27:11.190 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../040-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-24T08:27:11.234 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-24T08:27:11.541 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-tempita. 2023-11-24T08:27:11.568 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-24T08:27:11.608 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-24T08:27:11.911 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-paste. 2023-11-24T08:27:11.937 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../042-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-24T08:27:11.977 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-24T08:27:12.272 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-24T08:27:12.299 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-24T08:27:12.339 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-24T08:27:12.432 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-base. 2023-11-24T08:27:12.459 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../023-ceph-base_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:12.508 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-base (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:12.642 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-24T08:27:12.668 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-24T08:27:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-24T08:27:13.020 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-webtest. 2023-11-24T08:27:13.047 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../045-python3-webtest_2.0.35-1_all.deb ... 2023-11-24T08:27:13.086 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-24T08:27:13.170 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-24T08:27:13.185 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../024-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-24T08:27:13.220 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-24T08:27:13.389 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pecan. 2023-11-24T08:27:13.416 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-24T08:27:13.456 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-24T08:27:13.514 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cheroot. 2023-11-24T08:27:13.541 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../025-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-24T08:27:13.581 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-24T08:27:13.796 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-certifi. 2023-11-24T08:27:13.823 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../047-python3-certifi_2020.6.20-1_all.deb ... 2023-11-24T08:27:13.862 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-24T08:27:13.909 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-24T08:27:13.935 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../026-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-24T08:27:13.975 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-24T08:27:14.207 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-idna. 2023-11-24T08:27:14.234 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../048-python3-idna_3.3-1_all.deb ... 2023-11-24T08:27:14.273 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-24T08:27:14.278 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-24T08:27:14.305 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../027-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-24T08:27:14.345 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-24T08:27:14.576 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-urllib3. 2023-11-24T08:27:14.603 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../049-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-11-24T08:27:14.606 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-24T08:27:14.621 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../028-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-24T08:27:14.643 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-11-24T08:27:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-24T08:27:14.959 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempora. 2023-11-24T08:27:14.971 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-requests. 2023-11-24T08:27:14.985 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../029-python3-tempora_4.1.2-1_all.deb ... 2023-11-24T08:27:14.998 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../050-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-24T08:27:15.025 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-24T08:27:15.038 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-24T08:27:15.324 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-24T08:27:15.336 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-portend. 2023-11-24T08:27:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../051-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-24T08:27:15.363 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../030-python3-portend_3.0.0-1_all.deb ... 2023-11-24T08:27:15.373 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-24T08:27:15.403 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-24T08:27:15.668 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-24T08:27:15.681 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-24T08:27:15.684 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../052-ceph-mgr-modules-core_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:15.708 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../031-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-24T08:27:15.726 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:15.739 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-24T08:27:16.026 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-24T08:27:16.052 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../032-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-24T08:27:16.081 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-24T08:27:16.092 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-24T08:27:16.108 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../053-libsqlite3-mod-ceph_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:16.148 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:16.438 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-natsort. 2023-11-24T08:27:16.442 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-openssl. 2023-11-24T08:27:16.464 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../033-python3-natsort_8.0.2-1_all.deb ... 2023-11-24T08:27:16.469 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../054-python3-openssl_21.0.0-1_all.deb ... 2023-11-24T08:27:16.504 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-24T08:27:16.508 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-24T08:27:16.790 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-logutils. 2023-11-24T08:27:16.817 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../034-python3-logutils_0.3.3-8_all.deb ... 2023-11-24T08:27:16.845 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr. 2023-11-24T08:27:16.857 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-24T08:27:16.871 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../055-ceph-mgr_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:16.911 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:17.390 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mon. 2023-11-24T08:27:17.416 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../056-ceph-mon_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:17.448 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mon (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:17.777 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-24T08:27:17.803 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../035-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-24T08:27:17.843 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-24T08:27:18.155 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-mako. 2023-11-24T08:27:18.181 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../036-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-24T08:27:18.221 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-24T08:27:18.335 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-osd. 2023-11-24T08:27:18.361 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../057-ceph-osd_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:18.401 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-osd (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:18.550 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-24T08:27:18.576 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../037-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-24T08:27:18.616 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-24T08:27:18.886 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-24T08:27:18.912 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../038-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-24T08:27:18.952 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-24T08:27:19.413 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webob. 2023-11-24T08:27:19.440 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../039-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-24T08:27:19.480 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-24T08:27:19.841 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-waitress. 2023-11-24T08:27:19.868 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../040-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-24T08:27:19.911 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-24T08:27:20.219 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempita. 2023-11-24T08:27:20.245 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-24T08:27:20.285 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-24T08:27:20.513 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph. 2023-11-24T08:27:20.528 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../058-ceph_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:20.571 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:20.588 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-paste. 2023-11-24T08:27:20.615 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../042-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-24T08:27:20.655 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-24T08:27:20.840 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-fuse. 2023-11-24T08:27:20.867 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../059-ceph-fuse_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:20.898 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-fuse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:20.950 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-24T08:27:20.977 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-24T08:27:21.016 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-24T08:27:21.260 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mds. 2023-11-24T08:27:21.269 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-24T08:27:21.284 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-24T08:27:21.287 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../060-ceph-mds_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:21.319 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-24T08:27:21.351 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mds (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:21.631 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webtest. 2023-11-24T08:27:21.657 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../045-python3-webtest_2.0.35-1_all.deb ... 2023-11-24T08:27:21.697 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-24T08:27:21.738 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package cephadm. 2023-11-24T08:27:21.765 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../061-cephadm_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:21.805 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking cephadm (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:22.008 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pecan. 2023-11-24T08:27:22.035 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-24T08:27:22.075 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-24T08:27:22.090 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-24T08:27:22.106 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../062-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-24T08:27:22.140 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-24T08:27:22.438 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-certifi. 2023-11-24T08:27:22.443 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-24T08:27:22.459 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../063-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-24T08:27:22.464 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../047-python3-certifi_2020.6.20-1_all.deb ... 2023-11-24T08:27:22.501 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-24T08:27:22.504 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-24T08:27:22.808 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-idna. 2023-11-24T08:27:22.834 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../048-python3-idna_3.3-1_all.deb ... 2023-11-24T08:27:22.899 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-24T08:27:23.152 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-urllib3. 2023-11-24T08:27:23.175 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../049-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-11-24T08:27:23.218 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-11-24T08:27:23.514 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests. 2023-11-24T08:27:23.532 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-babel. 2023-11-24T08:27:23.540 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../050-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-24T08:27:23.559 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../064-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-24T08:27:23.580 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-24T08:27:23.598 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-24T08:27:23.891 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-24T08:27:23.918 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../051-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-24T08:27:23.934 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-jinja2. 2023-11-24T08:27:23.950 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../065-python3-jinja2_3.0.3-1_all.deb ... 2023-11-24T08:27:23.957 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-24T08:27:23.993 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-24T08:27:24.219 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-24T08:27:24.234 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../052-ceph-mgr-modules-core_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:24.277 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:24.338 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-24T08:27:24.365 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:24.404 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:24.642 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-24T08:27:24.669 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../053-libsqlite3-mod-ceph_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:24.700 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:24.749 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-24T08:27:24.775 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../067-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-24T08:27:24.816 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-24T08:27:24.987 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-openssl. 2023-11-24T08:27:25.013 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../054-python3-openssl_21.0.0-1_all.deb ... 2023-11-24T08:27:25.053 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-24T08:27:25.160 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-24T08:27:25.187 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../068-python3-repoze.lru_0.7-2_all.deb ... 2023-11-24T08:27:25.227 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-24T08:27:25.372 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr. 2023-11-24T08:27:25.399 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../055-ceph-mgr_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:25.439 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:25.513 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-routes. 2023-11-24T08:27:25.540 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../069-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-24T08:27:25.580 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-24T08:27:25.800 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mon. 2023-11-24T08:27:25.827 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../056-ceph-mon_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:25.858 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mon (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:25.908 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-24T08:27:25.935 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:25.974 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:26.353 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-24T08:27:26.369 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../071-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-24T08:27:26.411 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-24T08:27:26.820 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-osd. 2023-11-24T08:27:26.847 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../057-ceph-osd_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:26.887 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-osd (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:26.889 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-joblib. 2023-11-24T08:27:26.917 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../072-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-24T08:27:26.956 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-24T08:27:27.318 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-24T08:27:27.345 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../073-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-24T08:27:27.385 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-24T08:27:27.687 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-sklearn. 2023-11-24T08:27:27.714 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../074-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-24T08:27:27.962 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-24T08:27:28.662 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-24T08:27:28.690 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../075-ceph-mgr-diskprediction-local_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:28.698 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph. 2023-11-24T08:27:28.725 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../058-ceph_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:28.729 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:28.764 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:29.059 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-fuse. 2023-11-24T08:27:29.086 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../059-ceph-fuse_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:29.125 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-fuse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:29.495 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mds. 2023-11-24T08:27:29.523 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../060-ceph-mds_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:29.649 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-cachetools. 2023-11-24T08:27:29.670 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mds (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:29.676 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../076-python3-cachetools_5.0.0-1_all.deb ... 2023-11-24T08:27:29.707 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-24T08:27:29.976 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-24T08:27:30.004 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../077-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-24T08:27:30.042 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-24T08:27:30.057 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package cephadm. 2023-11-24T08:27:30.083 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../061-cephadm_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:30.123 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking cephadm (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:30.388 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-24T08:27:30.416 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../078-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-24T08:27:30.417 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-24T08:27:30.444 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../062-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-24T08:27:30.454 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-24T08:27:30.484 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-24T08:27:30.749 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-rsa. 2023-11-24T08:27:30.765 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../079-python3-rsa_4.8-1_all.deb ... 2023-11-24T08:27:30.795 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-24T08:27:30.799 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-24T08:27:30.811 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../063-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-24T08:27:30.853 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-24T08:27:31.111 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-google-auth. 2023-11-24T08:27:31.138 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../080-python3-google-auth_1.5.1-3_all.deb ... 2023-11-24T08:27:31.177 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-24T08:27:31.505 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-24T08:27:31.532 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../081-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-24T08:27:31.572 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-24T08:27:31.833 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-babel. 2023-11-24T08:27:31.860 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../064-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-24T08:27:31.883 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-websocket. 2023-11-24T08:27:31.900 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-24T08:27:31.910 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../082-python3-websocket_1.2.3-1_all.deb ... 2023-11-24T08:27:31.949 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-24T08:27:32.228 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jinja2. 2023-11-24T08:27:32.254 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../065-python3-jinja2_3.0.3-1_all.deb ... 2023-11-24T08:27:32.294 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-24T08:27:32.379 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-24T08:27:32.406 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../083-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-24T08:27:32.454 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-24T08:27:32.614 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-24T08:27:32.641 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:32.681 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:33.034 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-24T08:27:33.061 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../067-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-24T08:27:33.101 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-24T08:27:33.117 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-24T08:27:33.133 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../084-ceph-mgr-k8sevents_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:33.167 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:33.404 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-24T08:27:33.430 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../068-python3-repoze.lru_0.7-2_all.deb ... 2023-11-24T08:27:33.470 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-24T08:27:33.487 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-24T08:27:33.514 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../085-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-24T08:27:33.554 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-24T08:27:33.774 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-routes. 2023-11-24T08:27:33.800 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../069-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-24T08:27:33.839 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-24T08:27:33.899 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-24T08:27:33.926 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../086-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-24T08:27:33.965 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-24T08:27:34.176 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-24T08:27:34.202 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:34.209 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package jq. 2023-11-24T08:27:34.236 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../087-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-24T08:27:34.242 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:34.267 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-24T08:27:34.495 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package socat. 2023-11-24T08:27:34.510 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../088-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-24T08:27:34.553 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-24T08:27:34.646 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-24T08:27:34.661 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../071-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-24T08:27:34.703 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-24T08:27:34.848 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package xmlstarlet. 2023-11-24T08:27:34.864 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../089-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-24T08:27:34.906 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-24T08:27:35.199 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-joblib. 2023-11-24T08:27:35.226 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../072-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-24T08:27:35.252 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-test. 2023-11-24T08:27:35.265 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-24T08:27:35.267 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../090-ceph-test_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:35.310 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-test (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:35.611 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-24T08:27:35.626 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../073-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-24T08:27:35.660 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-24T08:27:35.938 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn. 2023-11-24T08:27:35.965 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../074-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-24T08:27:36.005 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-24T08:27:36.692 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-24T08:27:36.720 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../075-ceph-mgr-diskprediction-local_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:36.763 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:37.497 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package ceph-volume. 2023-11-24T08:27:37.524 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../091-ceph-volume_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:37.563 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking ceph-volume (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:37.699 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cachetools. 2023-11-24T08:27:37.726 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../076-python3-cachetools_5.0.0-1_all.deb ... 2023-11-24T08:27:37.766 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-24T08:27:37.859 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-24T08:27:37.875 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../092-libcephfs-dev_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:37.917 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking libcephfs-dev (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:38.035 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-24T08:27:38.051 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../077-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-24T08:27:38.085 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-24T08:27:38.345 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package nvme-cli. 2023-11-24T08:27:38.373 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../093-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-24T08:27:38.388 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-24T08:27:38.415 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../078-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-24T08:27:38.453 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-24T08:27:38.455 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-24T08:27:38.850 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rsa. 2023-11-24T08:27:38.877 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../079-python3-rsa_4.8-1_all.deb ... 2023-11-24T08:27:38.900 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-24T08:27:38.916 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-24T08:27:38.917 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-24T08:27:38.958 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-24T08:27:39.236 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-google-auth. 2023-11-24T08:27:39.263 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../080-python3-google-auth_1.5.1-3_all.deb ... 2023-11-24T08:27:39.330 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-24T08:27:39.362 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-24T08:27:39.389 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-24T08:27:39.428 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-24T08:27:39.631 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-24T08:27:39.658 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../081-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-24T08:27:39.697 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-24T08:27:39.731 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pastescript. 2023-11-24T08:27:39.759 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2023-11-24T08:27:39.797 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-24T08:27:39.975 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-websocket. 2023-11-24T08:27:40.002 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../082-python3-websocket_1.2.3-1_all.deb ... 2023-11-24T08:27:40.033 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-24T08:27:40.118 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pluggy. 2023-11-24T08:27:40.146 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-24T08:27:40.184 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-24T08:27:40.404 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-24T08:27:40.431 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../083-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-24T08:27:40.471 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-24T08:27:40.495 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-psutil. 2023-11-24T08:27:40.522 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-24T08:27:40.561 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-24T08:27:40.906 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-py. 2023-11-24T08:27:40.934 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2023-11-24T08:27:40.981 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-24T08:27:41.143 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-24T08:27:41.171 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../084-ceph-mgr-k8sevents_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:41.209 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:41.318 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pygments. 2023-11-24T08:27:41.346 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-24T08:27:41.384 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-24T08:27:41.496 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-24T08:27:41.523 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../085-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-24T08:27:41.554 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-24T08:27:41.830 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-24T08:27:41.858 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-24T08:27:41.890 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-24T08:27:41.896 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-24T08:27:41.918 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../086-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-24T08:27:41.956 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-24T08:27:42.174 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-toml. 2023-11-24T08:27:42.201 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2023-11-24T08:27:42.217 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package jq. 2023-11-24T08:27:42.240 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-24T08:27:42.245 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../087-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-24T08:27:42.284 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-24T08:27:42.520 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package socat. 2023-11-24T08:27:42.543 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-pytest. 2023-11-24T08:27:42.547 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../088-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-24T08:27:42.571 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-24T08:27:42.586 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-24T08:27:42.609 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-24T08:27:42.864 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package xmlstarlet. 2023-11-24T08:27:42.881 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../089-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-24T08:27:42.914 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-24T08:27:42.921 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package python3-simplejson. 2023-11-24T08:27:42.949 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-24T08:27:42.996 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-24T08:27:43.227 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-test. 2023-11-24T08:27:43.240 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-24T08:27:43.243 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../090-ceph-test_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:43.268 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-24T08:27:43.285 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-test (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:43.307 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-24T08:27:44.036 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package radosgw. 2023-11-24T08:27:44.064 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../106-radosgw_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:44.102 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking radosgw (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:44.855 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package rbd-fuse. 2023-11-24T08:27:44.871 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../107-rbd-fuse_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:44.913 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking rbd-fuse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:45.241 INFO:teuthology.orchestra.run.smithi164.stdout:Selecting previously unselected package rbd-mirror. 2023-11-24T08:27:45.269 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing to unpack .../108-rbd-mirror_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:45.307 INFO:teuthology.orchestra.run.smithi164.stdout:Unpacking rbd-mirror (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:45.622 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-volume. 2023-11-24T08:27:45.649 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../091-ceph-volume_18.0.0-7484-g135a1ac9-1jammy_all.deb ... 2023-11-24T08:27:45.688 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-volume (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:45.770 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-24T08:27:45.992 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-24T08:27:46.008 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../092-libcephfs-dev_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:46.012 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-24T08:27:46.042 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs-dev (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:46.139 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-24T08:27:46.354 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package nvme-cli. 2023-11-24T08:27:46.381 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../093-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-24T08:27:46.397 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-24T08:27:46.411 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-24T08:27:46.742 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-24T08:27:46.769 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-24T08:27:46.774 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-24T08:27:46.808 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-24T08:27:47.220 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-24T08:27:47.244 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-24T08:27:47.265 INFO:teuthology.orchestra.run.smithi164.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-24T08:27:47.278 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-24T08:27:47.366 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up cephadm (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:47.547 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastescript. 2023-11-24T08:27:47.575 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2023-11-24T08:27:47.614 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-24T08:27:47.884 INFO:teuthology.orchestra.run.smithi164.stdout:Adding system user cephadm....done 2023-11-24T08:27:47.934 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pluggy. 2023-11-24T08:27:47.961 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-24T08:27:48.000 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-24T08:27:48.046 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-24T08:27:48.312 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-psutil. 2023-11-24T08:27:48.314 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-24T08:27:48.339 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-24T08:27:48.378 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-24T08:27:48.565 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-24T08:27:48.692 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-24T08:27:48.698 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-py. 2023-11-24T08:27:48.725 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2023-11-24T08:27:48.764 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-24T08:27:48.943 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-24T08:27:49.068 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pygments. 2023-11-24T08:27:49.092 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-24T08:27:49.126 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-24T08:27:49.210 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-24T08:27:49.311 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-24T08:27:49.588 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-24T08:27:49.613 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-24T08:27:49.616 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-24T08:27:49.655 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-24T08:27:49.957 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-toml. 2023-11-24T08:27:49.973 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-24T08:27:49.985 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2023-11-24T08:27:50.024 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-24T08:27:50.249 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-24T08:27:50.327 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pytest. 2023-11-24T08:27:50.354 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-24T08:27:50.393 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-24T08:27:50.508 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-24T08:27:50.730 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplejson. 2023-11-24T08:27:50.734 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-24T08:27:50.746 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-24T08:27:50.780 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-24T08:27:50.969 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-24T08:27:51.016 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-24T08:27:51.032 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-24T08:27:51.066 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-24T08:27:51.220 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-ceph-argparse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:51.479 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-24T08:27:51.745 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package radosgw. 2023-11-24T08:27:51.764 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-24T08:27:51.772 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../106-radosgw_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:51.803 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking radosgw (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:52.023 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-24T08:27:52.150 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-24T08:27:52.500 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-24T08:27:52.522 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package rbd-fuse. 2023-11-24T08:27:52.538 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../107-rbd-fuse_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:52.580 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking rbd-fuse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:52.610 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-24T08:27:52.870 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-24T08:27:52.917 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package rbd-mirror. 2023-11-24T08:27:52.944 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../108-rbd-mirror_18.0.0-7484-g135a1ac9-1jammy_amd64.deb ... 2023-11-24T08:27:52.983 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking rbd-mirror (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:53.153 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-24T08:27:53.420 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-24T08:27:53.456 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-24T08:27:53.688 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-24T08:27:53.806 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-24T08:27:54.051 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-24T08:27:54.098 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-24T08:27:54.374 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-24T08:27:54.407 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-24T08:27:54.475 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-24T08:27:54.576 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-24T08:27:54.861 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-24T08:27:54.947 INFO:teuthology.orchestra.run.smithi153.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-24T08:27:55.057 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up cephadm (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:55.245 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-24T08:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-24T08:27:55.642 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user cephadm....done 2023-11-24T08:27:55.788 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-24T08:27:55.890 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-24T08:27:56.073 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-24T08:27:56.190 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-24T08:27:56.324 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-24T08:27:56.450 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-24T08:27:56.517 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-24T08:27:56.701 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-24T08:27:56.751 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-24T08:27:56.869 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-ceph-common (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:56.952 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-24T08:27:57.053 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-24T08:27:57.182 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-idna (3.3-1) ... 2023-11-24T08:27:57.346 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-24T08:27:57.505 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-24T08:27:57.706 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-24T08:27:57.966 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-24T08:27:58.250 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-24T08:27:58.493 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-24T08:27:58.602 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:27:58.735 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-24T08:27:58.736 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-24T08:27:58.969 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-argparse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:27:59.229 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-24T08:27:59.299 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-24T08:27:59.399 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-24T08:27:59.430 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-24T08:27:59.516 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-11-24T08:27:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-24T08:27:59.792 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-24T08:27:59.849 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-24T08:28:00.035 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-24T08:28:00.208 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-24T08:28:00.278 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:28:00.334 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-24T08:28:00.404 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-24T08:28:00.697 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-24T08:28:00.794 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-24T08:28:00.973 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-24T08:28:01.070 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-24T08:28:01.232 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-24T08:28:01.388 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-24T08:28:01.475 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-24T08:28:01.717 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-24T08:28:01.985 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-24T08:28:02.072 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-24T08:28:02.111 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-24T08:28:02.340 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-24T08:28:02.387 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-24T08:28:02.458 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-24T08:28:02.513 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-24T08:28:02.575 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-24T08:28:02.814 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-24T08:28:02.860 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-24T08:28:03.115 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-24T08:28:03.236 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-24T08:28:03.358 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-24T08:28:03.487 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-24T08:28:03.754 INFO:teuthology.orchestra.run.smithi164.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-24T08:28:03.855 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-24T08:28:03.986 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-24T08:28:04.120 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-24T08:28:04.148 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-24T08:28:04.483 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-24T08:28:04.504 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-24T08:28:04.709 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-24T08:28:04.764 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-24T08:28:04.826 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-common (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:05.015 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:28:05.111 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-idna (3.3-1) ... 2023-11-24T08:28:05.124 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-24T08:28:05.437 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-24T08:28:05.442 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-24T08:28:05.735 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-24T08:28:06.003 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-24T08:28:06.129 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-24T08:28:06.396 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-24T08:28:06.601 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:28:06.664 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-24T08:28:06.726 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-24T08:28:06.924 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-24T08:28:06.961 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-24T08:28:07.087 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-24T08:28:07.213 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-11-24T08:28:07.292 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-24T08:28:07.514 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-24T08:28:07.560 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-24T08:28:07.767 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-24T08:28:07.895 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-24T08:28:08.010 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:28:08.071 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librados2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:08.119 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-24T08:28:08.197 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librgw2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:08.323 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:08.437 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-24T08:28:08.441 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libcephfs2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:08.558 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libradosstriper1 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:08.659 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-24T08:28:08.705 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-24T08:28:08.981 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-24T08:28:09.013 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up librbd1 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:09.138 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-modules-core (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:09.232 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-24T08:28:09.273 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-fuse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:09.474 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-24T08:28:09.487 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-24T08:28:09.487 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-24T08:28:09.726 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-24T08:28:09.852 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-24T08:28:10.008 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up libcephfs-dev (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:10.109 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rados (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:10.144 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-24T08:28:10.209 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-24T08:28:10.270 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-24T08:28:10.555 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-24T08:28:10.856 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-24T08:28:11.115 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-24T08:28:11.430 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rbd (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:11.494 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-24T08:28:11.582 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up rbd-fuse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:11.691 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-rgw (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:11.751 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-24T08:28:11.792 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up python3-cephfs (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:11.852 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-24T08:28:11.893 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-common (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:12.237 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-24T08:28:12.435 INFO:teuthology.orchestra.run.smithi164.stdout:Adding group ceph....done 2023-11-24T08:28:12.496 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-24T08:28:12.756 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-24T08:28:12.804 INFO:teuthology.orchestra.run.smithi164.stdout:Adding system user ceph....done 2023-11-24T08:28:12.882 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-24T08:28:12.980 INFO:teuthology.orchestra.run.smithi164.stdout:Setting system user ceph properties....done 2023-11-24T08:28:12.998 INFO:teuthology.orchestra.run.smithi164.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-24T08:28:13.134 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-24T08:28:13.208 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-24T08:28:13.465 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-24T08:28:13.500 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-24T08:28:13.793 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-24T08:28:13.919 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-24T08:28:13.986 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-test (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:14.195 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-24T08:28:14.395 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up radosgw (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:14.481 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-24T08:28:14.839 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-24T08:28:15.017 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-24T08:28:15.017 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-24T08:28:15.240 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-24T08:28:15.523 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-base (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:15.525 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-24T08:28:15.854 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-24T08:28:16.004 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-24T08:28:16.195 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librados2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:16.321 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librgw2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:16.374 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mds (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:16.446 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:16.572 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs2 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:16.601 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-24T08:28:16.602 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-24T08:28:16.698 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libradosstriper1 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:16.816 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-24T08:28:17.143 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up rbd-mirror (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:17.186 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librbd1 (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:17.312 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-modules-core (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:17.352 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-11-24T08:28:17.352 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-11-24T08:28:17.438 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-fuse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:17.644 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-24T08:28:17.644 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-24T08:28:17.860 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:18.098 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-24T08:28:18.099 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-24T08:28:18.182 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs-dev (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:18.282 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rados (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:18.400 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-24T08:28:18.662 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-osd (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:19.098 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-24T08:28:19.099 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-24T08:28:19.622 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:19.663 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rbd (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:19.772 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up rbd-fuse (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:19.873 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rgw (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:19.974 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cephfs (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:19.975 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-common (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:20.142 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mon (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:20.354 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-24T08:28:20.355 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-24T08:28:20.626 INFO:teuthology.orchestra.run.smithi153.stdout:Adding group ceph....done 2023-11-24T08:28:20.885 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-cephadm (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:20.969 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user ceph....done 2023-11-24T08:28:21.036 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:21.112 INFO:teuthology.orchestra.run.smithi153.stdout:Setting system user ceph properties....done 2023-11-24T08:28:21.129 INFO:teuthology.orchestra.run.smithi153.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-24T08:28:21.154 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-mgr-dashboard (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:21.263 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-24T08:28:21.279 INFO:teuthology.orchestra.run.smithi164.stdout:Setting up ceph-volume (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:21.607 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-24T08:28:21.615 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-24T08:28:22.152 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-test (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:22.278 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up radosgw (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:22.408 INFO:teuthology.orchestra.run.smithi164.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-24T08:28:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-24T08:28:22.870 DEBUG:teuthology.parallel:result is None 2023-11-24T08:28:23.098 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-24T08:28:23.098 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-24T08:28:23.605 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-base (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:23.927 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-24T08:28:24.418 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mds (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:24.729 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-24T08:28:24.729 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-24T08:28:25.276 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up rbd-mirror (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:25.487 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-11-24T08:28:25.487 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-11-24T08:28:26.002 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:26.205 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-24T08:28:26.205 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-24T08:28:26.849 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-osd (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:27.167 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-24T08:28:27.167 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-24T08:28:27.692 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:27.819 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:27.961 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mon (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:28.178 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-24T08:28:28.178 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-24T08:28:28.695 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-cephadm (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:28.838 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:28.939 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-dashboard (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:29.074 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-volume (18.0.0-7484-g135a1ac9-1jammy) ... 2023-11-24T08:28:29.409 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-24T08:28:30.126 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-24T08:28:30.575 INFO:teuthology.orchestra.run.smithi153.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-24T08:28:30.578 DEBUG:teuthology.parallel:result is None 2023-11-24T08:28:30.578 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 2023-11-24T08:28:30.717 DEBUG:teuthology.orchestra.run.smithi153:> dpkg-query -W -f '${Version}' ceph 2023-11-24T08:28:30.736 INFO:teuthology.orchestra.run.smithi153.stdout:18.0.0-7484-g135a1ac9-1jammy 2023-11-24T08:28:30.736 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7484-g135a1ac9-1jammy 2023-11-24T08:28:30.736 INFO:teuthology.task.install:The correct ceph version 18.0.0-7484-g135a1ac9-1jammy is installed. 2023-11-24T08:28:30.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 2023-11-24T08:28:30.951 DEBUG:teuthology.orchestra.run.smithi164:> dpkg-query -W -f '${Version}' ceph 2023-11-24T08:28:30.970 INFO:teuthology.orchestra.run.smithi164.stdout:18.0.0-7484-g135a1ac9-1jammy 2023-11-24T08:28:30.970 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7484-g135a1ac9-1jammy 2023-11-24T08:28:30.971 INFO:teuthology.task.install:The correct ceph version 18.0.0-7484-g135a1ac9-1jammy is installed. 2023-11-24T08:28:30.971 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-24T08:28:30.971 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:30.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-24T08:28:30.985 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:30.985 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-24T08:28:31.026 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-24T08:28:31.026 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:31.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2023-11-24T08:28:31.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-24T08:28:31.093 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:31.093 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2023-11-24T08:28:31.106 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-24T08:28:31.162 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-24T08:28:31.162 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:31.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-24T08:28:31.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-24T08:28:31.229 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:31.230 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-24T08:28:31.242 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-24T08:28:31.298 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-24T08:28:31.298 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:31.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2023-11-24T08:28:31.310 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-24T08:28:31.366 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:31.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2023-11-24T08:28:31.378 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-24T08:28:31.433 INFO:teuthology.run_tasks:Running task ceph... 2023-11-24T08:28:31.497 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-24T08:28:31.497 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2023-11-24T08:28:31.500 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2023-11-24T08:28:31.512 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-24T08:28:31.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-24T08:28:31.556 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-24T08:28:31.568 INFO:tasks.ceph:Creating extra log directories... 2023-11-24T08:28:31.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-24T08:28:31.608 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-24T08:28:31.635 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-11-24T08:28:31.635 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-24T08:28:31.635 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7465501', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2661, 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1', 'sleep_before_teardown': 0, 'subset': '14/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1', 'targets': {'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKay/u69gbSDH+wQg1rANAqFcLSHi3oa5yrvt7QTzuCp6Tnk1XI0S9vKeGRUUVuR7yn7Ern95TqUza/7zUGQSdo=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNN4flvWtU6IA2MNQJD2lLFhHIgd/PZ0iehpJcUsE7kQw/8ZZUYwpfu5sODRLPlVm8TccYiZoQyDcPZAMHRMrk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '2442239d2653456406c25ae0c71b689c8f2657b6', 'timestamp': '2023-11-24_05:31:39', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-24T08:28:31.636 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-11-24T08:28:31.664 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-11-24T08:28:31.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T08:28:31.712 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T08:28:31.726 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:31.726 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2023-11-24T08:28:31.772 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-24T08:28:31.772 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2023-11-24T08:28:31.820 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-24T08:28:31.820 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:28:31.820 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 889 Links: 1 2023-11-24T08:28:31.820 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:28:31.820 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-11-24 08:23:11.414857609 +0000 2023-11-24T08:28:31.820 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-11-24 08:23:11.062860753 +0000 2023-11-24T08:28:31.820 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-11-24 08:23:11.062860753 +0000 2023-11-24T08:28:31.820 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-11-24T08:28:31.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-24T08:28:31.874 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-11-24T08:28:31.874 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-11-24T08:28:31.874 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000385249 s, 1.3 MB/s 2023-11-24T08:28:31.875 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-24T08:28:31.924 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2023-11-24T08:28:31.971 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-24T08:28:31.971 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:28:31.971 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 899 Links: 1 2023-11-24T08:28:31.971 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:28:31.971 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-11-24 08:23:12.034852071 +0000 2023-11-24T08:28:31.971 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-11-24 08:23:11.666855358 +0000 2023-11-24T08:28:31.971 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-11-24 08:23:11.666855358 +0000 2023-11-24T08:28:31.972 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-11-24T08:28:31.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-24T08:28:32.025 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-11-24T08:28:32.026 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-11-24T08:28:32.026 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000412007 s, 1.2 MB/s 2023-11-24T08:28:32.027 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-24T08:28:32.080 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2023-11-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 918 Links: 1 2023-11-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:28:32.127 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-11-24 08:23:12.622846818 +0000 2023-11-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-11-24 08:23:12.274849927 +0000 2023-11-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-11-24 08:23:12.274849927 +0000 2023-11-24T08:28:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-11-24T08:28:32.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-24T08:28:32.181 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-11-24T08:28:32.182 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-11-24T08:28:32.182 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000389677 s, 1.3 MB/s 2023-11-24T08:28:32.183 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-24T08:28:32.237 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2023-11-24T08:28:32.283 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-24T08:28:32.283 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:28:32.283 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 932 Links: 1 2023-11-24T08:28:32.283 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:28:32.283 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-11-24 08:23:13.218841496 +0000 2023-11-24T08:28:32.283 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-11-24 08:23:12.850844782 +0000 2023-11-24T08:28:32.284 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-11-24 08:23:12.850844782 +0000 2023-11-24T08:28:32.284 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-11-24T08:28:32.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-24T08:28:32.338 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-11-24T08:28:32.338 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-11-24T08:28:32.338 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.0003924 s, 1.3 MB/s 2023-11-24T08:28:32.339 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-24T08:28:32.389 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-24T08:28:32.389 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-24T08:28:32.390 INFO:tasks.ceph:Generating config... 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client] debug rbd = 20 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client] log to stderr = False 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client] rbd default features = 125 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.392 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.393 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.393 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.393 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-11-24T08:28:32.393 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-11-24T08:28:32.393 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-11-24T08:28:32.393 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-24T08:28:32.393 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-24T08:28:32.393 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-24T08:28:32.393 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-24T08:28:32.393 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-24T08:28:32.393 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-24T08:28:32.393 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-24T08:28:32.393 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-24T08:28:32.394 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-24T08:28:32.395 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-24T08:28:32.395 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-24T08:28:32.395 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-11-24T08:28:32.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-11-24T08:28:32.465 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/cluster1.keyring 2023-11-24T08:28:32.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-11-24T08:28:32.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-11-24T08:28:32.601 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.153')] 2023-11-24T08:28:32.602 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.153', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-11-24T08:28:32.603 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:32.603 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T08:28:32.648 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.153 --print /home/ubuntu/cephtest/cluster1.monmap 2023-11-24T08:28:32.724 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-11-24T08:28:32.724 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: generated fsid 94e5c26f-c198-4207-b2c2-8bf1e80f7c68 2023-11-24T08:28:32.724 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = pacific 2023-11-24T08:28:32.725 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 0 2023-11-24T08:28:32.725 INFO:teuthology.orchestra.run.smithi153.stdout:fsid 94e5c26f-c198-4207-b2c2-8bf1e80f7c68 2023-11-24T08:28:32.725 INFO:teuthology.orchestra.run.smithi153.stdout:last_changed 2023-11-24T08:28:32.726022+0000 2023-11-24T08:28:32.725 INFO:teuthology.orchestra.run.smithi153.stdout:created 2023-11-24T08:28:32.726022+0000 2023-11-24T08:28:32.725 INFO:teuthology.orchestra.run.smithi153.stdout:min_mon_release 16 (pacific) 2023-11-24T08:28:32.725 INFO:teuthology.orchestra.run.smithi153.stdout:election_strategy: 1 2023-11-24T08:28:32.725 INFO:teuthology.orchestra.run.smithi153.stdout:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2023-11-24T08:28:32.725 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-11-24T08:28:32.726 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T08:28:32.779 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 94e5c26f-c198-4207-b2c2-8bf1e80f7c68... 2023-11-24T08:28:32.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-11-24T08:28:32.824 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.852 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.853 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2023-11-24T08:28:32.854 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = 172.21.15.153 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on pool no app = False 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 5000 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 94e5c26f-c198-4207-b2c2-8bf1e80f7c68 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2023-11-24T08:28:32.855 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.856 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: bdev async discard = True 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: bdev enable discard = True 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore allocator = stupid 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2023-11-24T08:28:32.857 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 1/20 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 1/20 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 4/10 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-11-24T08:28:32.858 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2023-11-24T08:28:32.859 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.860 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: debug rbd = 20 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: debug rbd_mirror = 15 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: log to stderr = False 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: rbd default features = 125 2023-11-24T08:28:32.861 INFO:teuthology.orchestra.run.smithi153.stdout: rbd mirror image policy type = simple 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.0] 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.1] 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.2] 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.3] 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.862 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.864 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2023-11-24T08:28:32.864 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2023-11-24T08:28:32.864 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T08:28:32.864 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2023-11-24T08:28:32.864 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.864 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2023-11-24T08:28:32.864 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.864 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2023-11-24T08:28:32.864 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T08:28:32.865 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T08:28:32.866 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = 172.21.15.153 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon client directed command retry = 5 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on pool no app = False 2023-11-24T08:28:32.867 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject socket failures = 5000 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 94e5c26f-c198-4207-b2c2-8bf1e80f7c68 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.868 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.869 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = stupid 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2023-11-24T08:28:32.870 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2023-11-24T08:28:32.871 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T08:28:32.872 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2023-11-24T08:28:32.873 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: debug rbd = 20 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: debug rbd_mirror = 15 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: log to stderr = False 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: rbd default features = 125 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: rbd mirror image policy type = simple 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.874 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.875 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2023-11-24T08:28:32.875 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.875 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.875 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2023-11-24T08:28:32.875 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:28:32.875 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:28:32.875 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-11-24T08:28:32.875 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-11-24T08:28:32.949 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-24T08:28:32.949 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:32.949 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-11-24T08:28:32.995 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:32.995 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-11-24T08:28:33.043 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T08:28:33.043 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:33.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/cluster1.keyring 2023-11-24T08:28:33.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-11-24T08:28:33.107 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:33.107 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-11-24T08:28:33.155 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T08:28:33.155 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:33.156 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/cluster1.keyring 2023-11-24T08:28:33.156 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-11-24T08:28:33.178 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:33.179 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-11-24T08:28:33.226 INFO:tasks.ceph:Setting up mon nodes... 2023-11-24T08:28:33.227 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-24T08:28:33.227 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-11-24T08:28:33.270 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-11-24T08:28:33.273 INFO:tasks.ceph:Setting up mds nodes... 2023-11-24T08:28:33.273 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-24T08:28:33.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-11-24T08:28:33.345 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-11-24T08:28:33.355 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-11-24T08:28:33.392 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-11-24T08:28:33.409 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-11-24T08:28:33.484 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-11-24T08:28:33.494 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-11-24T08:28:33.570 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-11-24T08:28:33.579 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-11-24T08:28:33.657 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-11-24T08:28:33.668 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-11-24T08:28:33.741 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-11-24T08:28:33.750 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-11-24T08:28:33.823 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-11-24T08:28:33.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-11-24T08:28:33.908 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-11-24T08:28:33.918 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-11-24T08:28:33.993 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-11-24T08:28:34.003 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-24T08:28:34.003 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-24T08:28:34.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-11-24T08:28:34.017 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-24T08:28:34.017 INFO:tasks.ceph:role: cluster1.osd.0 2023-11-24T08:28:34.017 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T08:28:34.017 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-24T08:28:34.078 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:28:34.078 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:28:34.078 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:28:34.078 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T08:28:34.078 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:28:34.078 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:28:34.079 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:28:34.079 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T08:28:34.079 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:28:34.079 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:28:34.569 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-11-24T08:28:34.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-11-24T08:28:34.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-11-24T08:28:34.721 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-11-24T08:28:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:28:34.733 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2023-11-24T08:28:34.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-11-24T08:28:34.786 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-24T08:28:34.787 INFO:tasks.ceph:role: cluster1.osd.1 2023-11-24T08:28:34.787 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T08:28:34.787 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-24T08:28:34.849 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:28:34.849 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:28:34.849 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:28:34.849 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T08:28:34.849 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:28:34.849 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:28:34.849 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:28:34.849 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T08:28:34.850 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:28:34.850 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:28:35.300 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-11-24T08:28:35.303 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-11-24T08:28:35.303 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-11-24T08:28:35.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-11-24T08:28:35.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:28:35.444 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2023-11-24T08:28:35.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-11-24T08:28:35.498 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-24T08:28:35.498 INFO:tasks.ceph:role: cluster1.osd.2 2023-11-24T08:28:35.498 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T08:28:35.498 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-24T08:28:35.563 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:28:35.563 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:28:35.563 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:28:35.563 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T08:28:35.563 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:28:35.563 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:28:35.563 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:28:35.564 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T08:28:35.564 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:28:35.564 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:28:36.028 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-11-24T08:28:36.030 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-11-24T08:28:36.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-11-24T08:28:36.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-11-24T08:28:36.186 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2023-11-24T08:28:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:28:36.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-11-24T08:28:36.301 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:36.297+0000 7fbd705a0640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-11-24T08:28:36.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:36.301+0000 7fbd705a0640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-11-24T08:28:36.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:36.301+0000 7fbd705a0640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-11-24T08:28:36.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:36.301+0000 7fbd705a0640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-11-24T08:28:36.306 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:36.301+0000 7fbd705a0640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-11-24T08:28:36.312 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:36.309+0000 7fbd705a0640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-11-24T08:28:39.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-11-24T08:28:39.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-11-24T08:28:39.418 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:39.413+0000 7f66f0942640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-11-24T08:28:39.418 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:39.417+0000 7f66f0942640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-11-24T08:28:39.419 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:39.417+0000 7f66f0942640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-11-24T08:28:39.419 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:39.417+0000 7f66f0942640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-11-24T08:28:39.419 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:39.417+0000 7f66f0942640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-11-24T08:28:39.419 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:39.417+0000 7f66f0942640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-11-24T08:28:42.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-11-24T08:28:42.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-11-24T08:28:42.573 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:42.569+0000 7fc37582a640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-11-24T08:28:42.573 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:42.569+0000 7fc37582a640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-11-24T08:28:42.573 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:42.569+0000 7fc37582a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-11-24T08:28:42.574 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:42.569+0000 7fc37582a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-11-24T08:28:42.574 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:42.569+0000 7fc37582a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-11-24T08:28:42.574 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-24T08:28:42.569+0000 7fc37582a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-11-24T08:28:45.582 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-11-24T08:28:45.639 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-24T08:28:45.639 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:45.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-11-24T08:28:45.695 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:45.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-11-24T08:28:45.750 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:45.750 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-11-24T08:28:45.806 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:45.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-11-24T08:28:45.862 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:45.863 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-11-24T08:28:45.911 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:45.911 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-11-24T08:28:45.917 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:45.917 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-11-24T08:28:45.966 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:45.966 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-11-24T08:28:46.014 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:46.015 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-11-24T08:28:46.062 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:46.062 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-11-24T08:28:46.110 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:46.110 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-11-24T08:28:46.158 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:46.158 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-11-24T08:28:46.206 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:28:46.206 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-11-24T08:28:46.254 INFO:tasks.ceph:Adding keys to all mons... 2023-11-24T08:28:46.254 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/cluster1.keyring 2023-11-24T08:28:46.266 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2023-11-24T08:28:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBl5uM1EBAAGSpWk4XjTs0xIbvjxglGDw== 2023-11-24T08:28:46.266 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2023-11-24T08:28:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQC0XmBlBFUUEhAAhDRw70mmcTp8hINKxccuDw== 2023-11-24T08:28:46.266 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQC3XmBleH8FGRAAZ/ZQTTcIIjAgHLdDmpY/xQ== 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQC6XmBlwns+IhAAL4uij0botUQ7fTruccsJCg== 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBlQpmyFBAAjCd5rHiDjdISE2Qm6tCRTA== 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror] 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBlyVt0FxAAGwZzgfhRCUgDTyUSeeF0jA== 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.0] 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBl2gf5HBAA9EBJINUJn4ekhd0nGmoufQ== 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.1] 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBl4z8QIhAA3a9reM8oKsxKBmN1L1M+/w== 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.2] 2023-11-24T08:28:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBlabBFJxAAPdxB+HC9M8hcmR8+ysxGTQ== 2023-11-24T08:28:46.268 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.3] 2023-11-24T08:28:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBlsqxCLBAA5qBnGHurZD+hEiXrAUafuw== 2023-11-24T08:28:46.268 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.4] 2023-11-24T08:28:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBlr+otMRAA23iWRODRcLG1A8X9cgYmtw== 2023-11-24T08:28:46.268 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.5] 2023-11-24T08:28:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBlAko5NhAAqtkbacu5sd209vBkwjh+PA== 2023-11-24T08:28:46.268 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.6] 2023-11-24T08:28:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCxXmBlq11DOxAAbWXZeGOxkyv8oPFhD47U+g== 2023-11-24T08:28:46.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T08:28:46.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:28:46.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:28:46.520 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:28:46.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:28:46.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:28:46.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:28:46.818 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:28:46.907 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:28:46.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:28:47.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:28:47.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:28:47.211 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:28:47.298 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-24T08:28:47.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-11-24T08:28:47.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-11-24T08:28:47.449 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-11-24T08:28:47.459 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-11-24T08:28:47.507 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-11-24T08:28:47.507 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-11-24T08:28:47.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-11-24T08:28:47.551 INFO:tasks.ceph.cluster1.mon.a:Started 2023-11-24T08:28:47.551 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-11-24T08:28:47.552 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-11-24T08:28:47.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-11-24T08:28:47.554 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-11-24T08:28:47.554 INFO:tasks.ceph:Setting crush tunables to default 2023-11-24T08:28:47.554 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster cluster1 osd crush tunables default 2023-11-24T08:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2023-11-24T08:28:47.835 INFO:tasks.ceph:check_enable_crimson: False 2023-11-24T08:28:47.836 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-11-24T08:28:47.836 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:47.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-11-24T08:28:47.848 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:47.848 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-11-24T08:28:47.905 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:28:47.906 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-11-24T08:28:47.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster cluster1 osd new 36b343bc-77eb-4879-93e5-2965f034bc29 0 2023-11-24T08:28:47.996 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:47.993+0000 7f1686a8c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-24T08:28:48.197 INFO:teuthology.orchestra.run.smithi153.stdout:0 2023-11-24T08:28:48.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster cluster1 osd new f5d4f77c-1f30-4418-aa0a-2760e5b4d966 1 2023-11-24T08:28:48.408 INFO:teuthology.orchestra.run.smithi153.stdout:1 2023-11-24T08:28:48.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster cluster1 osd new c0c18802-9220-4e2a-9ff6-81dc37fcdef8 2 2023-11-24T08:28:48.567 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:48.565+0000 7f1686a8c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-24T08:28:48.625 INFO:teuthology.orchestra.run.smithi153.stdout:2 2023-11-24T08:28:48.645 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-11-24T08:28:48.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-11-24T08:28:48.648 INFO:tasks.ceph.cluster1.osd.0:Started 2023-11-24T08:28:48.648 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-11-24T08:28:48.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-11-24T08:28:48.650 INFO:tasks.ceph.cluster1.osd.1:Started 2023-11-24T08:28:48.650 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-11-24T08:28:48.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-11-24T08:28:48.652 INFO:tasks.ceph.cluster1.osd.2:Started 2023-11-24T08:28:48.652 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-11-24T08:28:48.803 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:48.801+0000 7f1686a8c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-24T08:28:48.842 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:28:48.842 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":5,"fsid":"94e5c26f-c198-4207-b2c2-8bf1e80f7c68","created":"2023-11-24T08:28:47.691592+0000","modified":"2023-11-24T08:28:48.625743+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T08:28:48.625743+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36b343bc-77eb-4879-93e5-2965f034bc29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5d4f77c-1f30-4418-aa0a-2760e5b4d966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0c18802-9220-4e2a-9ff6-81dc37fcdef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:28:48.860 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-11-24T08:28:48.860 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-24T08:28:49.162 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-11-24T08:28:49.271 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T08:28:49.269+0000 7fe342cd8640 -1 Falling back to public interface 2023-11-24T08:28:49.272 INFO:tasks.ceph.cluster1.osd.1.smithi153.stderr:2023-11-24T08:28:49.269+0000 7f9989f3d640 -1 Falling back to public interface 2023-11-24T08:28:49.287 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:28:49.285+0000 7f4b6b607640 -1 Falling back to public interface 2023-11-24T08:28:49.345 INFO:teuthology.misc.health.smithi153.stdout: 2023-11-24T08:28:49.346 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":5,"fsid":"94e5c26f-c198-4207-b2c2-8bf1e80f7c68","created":"2023-11-24T08:28:47.691592+0000","modified":"2023-11-24T08:28:48.625743+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T08:28:48.625743+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36b343bc-77eb-4879-93e5-2965f034bc29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5d4f77c-1f30-4418-aa0a-2760e5b4d966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0c18802-9220-4e2a-9ff6-81dc37fcdef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:28:49.360 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-24T08:28:49.394 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:49.393+0000 7f1686a8c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-24T08:28:49.448 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:49.445+0000 7f1686a8c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-24T08:28:49.500 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:49.497+0000 7f1686a8c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-24T08:28:49.576 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:49.573+0000 7f1686a8c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-24T08:28:49.628 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:49.625+0000 7f1686a8c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-24T08:28:49.704 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:49.701+0000 7f1686a8c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-24T08:28:49.755 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:49.753+0000 7f1686a8c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-24T08:28:49.894 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:49.889+0000 7f1686a8c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-24T08:28:49.953 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:49.949+0000 7f1686a8c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-24T08:28:50.623 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-11-24T08:28:50.623 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-11-24T08:28:50.623 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr: from numpy import show_config as show_numpy_config 2023-11-24T08:28:50.696 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:50.693+0000 7f1686a8c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-24T08:28:50.865 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:50.861+0000 7f1686a8c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-24T08:28:51.030 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:51.025+0000 7f1686a8c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-24T08:28:51.099 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:51.097+0000 7f1686a8c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-24T08:28:51.235 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:51.233+0000 7f1686a8c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-24T08:28:51.843 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:51.841+0000 7f1686a8c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-24T08:28:51.917 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:51.913+0000 7f1686a8c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-24T08:28:52.094 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:52.089+0000 7f1686a8c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-24T08:28:52.166 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:52.165+0000 7f1686a8c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-24T08:28:52.495 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:52.493+0000 7f1686a8c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-24T08:28:52.555 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:52.553+0000 7f1686a8c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-24T08:28:53.108 INFO:tasks.ceph.cluster1.osd.1.smithi153.stderr:2023-11-24T08:28:53.105+0000 7f9989f3d640 -1 osd.1 0 log_to_monitors true 2023-11-24T08:28:53.145 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:28:53.141+0000 7f4b6b607640 -1 osd.2 0 log_to_monitors true 2023-11-24T08:28:53.192 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T08:28:53.189+0000 7fe342cd8640 -1 osd.0 0 log_to_monitors true 2023-11-24T08:28:53.592 INFO:tasks.ceph.cluster1.mgr.x.smithi153.stderr:2023-11-24T08:28:53.589+0000 7f162e312640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-24T08:28:55.663 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-11-24T08:28:56.084 INFO:teuthology.misc.health.smithi153.stdout: 2023-11-24T08:28:56.132 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":7,"fsid":"94e5c26f-c198-4207-b2c2-8bf1e80f7c68","created":"2023-11-24T08:28:47.691592+0000","modified":"2023-11-24T08:28:54.646382+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T08:28:48.625743+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36b343bc-77eb-4879-93e5-2965f034bc29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5d4f77c-1f30-4418-aa0a-2760e5b4d966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0c18802-9220-4e2a-9ff6-81dc37fcdef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:28:56.134 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-24T08:28:56.695 INFO:tasks.ceph.cluster1.osd.1.smithi153.stderr:2023-11-24T08:28:56.693+0000 7f9986efa640 -1 osd.1 0 waiting for initial osdmap 2023-11-24T08:28:56.722 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T08:28:56.717+0000 7fe33fc95640 -1 osd.0 0 waiting for initial osdmap 2023-11-24T08:28:56.723 INFO:tasks.ceph.cluster1.osd.1.smithi153.stderr:2023-11-24T08:28:56.721+0000 7f997dd11640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:28:56.751 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T08:28:56.749+0000 7fe336aac640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:28:56.755 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:28:56.753+0000 7f4b66d8e640 -1 osd.2 0 waiting for initial osdmap 2023-11-24T08:28:56.787 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:28:56.785+0000 7f4b5f3db640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:29:02.437 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-11-24T08:29:02.732 INFO:teuthology.misc.health.smithi153.stdout: 2023-11-24T08:29:02.732 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"94e5c26f-c198-4207-b2c2-8bf1e80f7c68","created":"2023-11-24T08:28:47.691592+0000","modified":"2023-11-24T08:29:00.777747+0000","last_up_change":"2023-11-24T08:28:57.738525+0000","last_in_change":"2023-11-24T08:28:48.625743+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:28:58.672360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"36b343bc-77eb-4879-93e5-2965f034bc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6801","nonce":3990165559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6805","nonce":3990165559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6815","nonce":3990165559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6810","nonce":3990165559}]},"public_addr":"172.21.15.153:6801/3990165559","cluster_addr":"172.21.15.153:6805/3990165559","heartbeat_back_addr":"172.21.15.153:6815/3990165559","heartbeat_front_addr":"172.21.15.153:6810/3990165559","state":["exists","up"]},{"osd":1,"uuid":"f5d4f77c-1f30-4418-aa0a-2760e5b4d966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6804","nonce":822903425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6808","nonce":822903425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6813","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6814","nonce":822903425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6811","nonce":822903425}]},"public_addr":"172.21.15.153:6804/822903425","cluster_addr":"172.21.15.153:6808/822903425","heartbeat_back_addr":"172.21.15.153:6814/822903425","heartbeat_front_addr":"172.21.15.153:6811/822903425","state":["exists","up"]},{"osd":2,"uuid":"c0c18802-9220-4e2a-9ff6-81dc37fcdef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6817","nonce":1195648817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6819","nonce":1195648817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6823","nonce":1195648817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6821","nonce":1195648817}]},"public_addr":"172.21.15.153:6817/1195648817","cluster_addr":"172.21.15.153:6819/1195648817","heartbeat_back_addr":"172.21.15.153:6823/1195648817","heartbeat_front_addr":"172.21.15.153:6821/1195648817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.207630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.127077+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.177065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:29:02.750 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-24T08:29:02.750 INFO:tasks.ceph:Creating RBD pool 2023-11-24T08:29:02.750 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-11-24T08:29:03.793 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'rbd' created 2023-11-24T08:29:03.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-24T08:29:04.799 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-24T08:29:04.829 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-11-24T08:29:04.829 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-11-24T08:29:04.830 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-11-24T08:29:04.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-11-24T08:29:04.830 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-24T08:29:05.158 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:05.174 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2353853796},{"type":"v1","addr":"172.21.15.153:6825","nonce":2353853796}]},"active_addr":"172.21.15.153:6825/2353853796","active_change":"2023-11-24T08:28:52.577329+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3628984519}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":2109496022}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3279002185}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":238891641}]}]} 2023-11-24T08:29:05.177 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-11-24T08:29:05.178 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-11-24T08:29:05.178 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-11-24T08:29:05.453 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:05.453 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":13,"fsid":"94e5c26f-c198-4207-b2c2-8bf1e80f7c68","created":"2023-11-24T08:28:47.691592+0000","modified":"2023-11-24T08:29:04.795199+0000","last_up_change":"2023-11-24T08:28:57.738525+0000","last_in_change":"2023-11-24T08:28:48.625743+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:28:58.672360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-24T08:29:03.081170+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"36b343bc-77eb-4879-93e5-2965f034bc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6801","nonce":3990165559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6805","nonce":3990165559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6815","nonce":3990165559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6810","nonce":3990165559}]},"public_addr":"172.21.15.153:6801/3990165559","cluster_addr":"172.21.15.153:6805/3990165559","heartbeat_back_addr":"172.21.15.153:6815/3990165559","heartbeat_front_addr":"172.21.15.153:6810/3990165559","state":["exists","up"]},{"osd":1,"uuid":"f5d4f77c-1f30-4418-aa0a-2760e5b4d966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6804","nonce":822903425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6808","nonce":822903425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6813","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6814","nonce":822903425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6811","nonce":822903425}]},"public_addr":"172.21.15.153:6804/822903425","cluster_addr":"172.21.15.153:6808/822903425","heartbeat_back_addr":"172.21.15.153:6814/822903425","heartbeat_front_addr":"172.21.15.153:6811/822903425","state":["exists","up"]},{"osd":2,"uuid":"c0c18802-9220-4e2a-9ff6-81dc37fcdef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6817","nonce":1195648817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6819","nonce":1195648817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6823","nonce":1195648817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6821","nonce":1195648817}]},"public_addr":"172.21.15.153:6817/1195648817","cluster_addr":"172.21.15.153:6819/1195648817","heartbeat_back_addr":"172.21.15.153:6823/1195648817","heartbeat_front_addr":"172.21.15.153:6821/1195648817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.207630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.127077+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.177065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:29:05.470 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-11-24T08:29:05.470 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-11-24T08:29:05.744 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:05.744 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":13,"fsid":"94e5c26f-c198-4207-b2c2-8bf1e80f7c68","created":"2023-11-24T08:28:47.691592+0000","modified":"2023-11-24T08:29:04.795199+0000","last_up_change":"2023-11-24T08:28:57.738525+0000","last_in_change":"2023-11-24T08:28:48.625743+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:28:58.672360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-24T08:29:03.081170+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"36b343bc-77eb-4879-93e5-2965f034bc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6801","nonce":3990165559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6805","nonce":3990165559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6815","nonce":3990165559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6810","nonce":3990165559}]},"public_addr":"172.21.15.153:6801/3990165559","cluster_addr":"172.21.15.153:6805/3990165559","heartbeat_back_addr":"172.21.15.153:6815/3990165559","heartbeat_front_addr":"172.21.15.153:6810/3990165559","state":["exists","up"]},{"osd":1,"uuid":"f5d4f77c-1f30-4418-aa0a-2760e5b4d966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6804","nonce":822903425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6808","nonce":822903425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6813","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6814","nonce":822903425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6811","nonce":822903425}]},"public_addr":"172.21.15.153:6804/822903425","cluster_addr":"172.21.15.153:6808/822903425","heartbeat_back_addr":"172.21.15.153:6814/822903425","heartbeat_front_addr":"172.21.15.153:6811/822903425","state":["exists","up"]},{"osd":2,"uuid":"c0c18802-9220-4e2a-9ff6-81dc37fcdef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6817","nonce":1195648817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6819","nonce":1195648817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6823","nonce":1195648817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6821","nonce":1195648817}]},"public_addr":"172.21.15.153:6817/1195648817","cluster_addr":"172.21.15.153:6819/1195648817","heartbeat_back_addr":"172.21.15.153:6823/1195648817","heartbeat_front_addr":"172.21.15.153:6821/1195648817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.207630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.127077+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.177065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:29:05.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-11-24T08:29:05.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-11-24T08:29:05.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-11-24T08:29:05.921 INFO:teuthology.orchestra.run.smithi153.stdout:34359738373 2023-11-24T08:29:05.921 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-11-24T08:29:05.925 INFO:teuthology.orchestra.run.smithi153.stdout:34359738374 2023-11-24T08:29:05.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-11-24T08:29:05.928 INFO:teuthology.orchestra.run.smithi153.stdout:34359738375 2023-11-24T08:29:05.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-11-24T08:29:06.196 INFO:teuthology.orchestra.run.smithi153.stdout:34359738372 2023-11-24T08:29:06.213 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2023-11-24T08:29:06.239 INFO:teuthology.orchestra.run.smithi153.stdout:34359738373 2023-11-24T08:29:06.246 INFO:teuthology.orchestra.run.smithi153.stdout:34359738374 2023-11-24T08:29:06.257 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.1 2023-11-24T08:29:06.264 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738374 for osd.0 2023-11-24T08:29:07.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-11-24T08:29:07.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-11-24T08:29:07.265 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-11-24T08:29:07.512 INFO:teuthology.orchestra.run.smithi153.stdout:34359738373 2023-11-24T08:29:07.529 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2023-11-24T08:29:07.529 DEBUG:teuthology.parallel:result is None 2023-11-24T08:29:07.549 INFO:teuthology.orchestra.run.smithi153.stdout:34359738374 2023-11-24T08:29:07.566 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.1 2023-11-24T08:29:07.566 DEBUG:teuthology.parallel:result is None 2023-11-24T08:29:07.584 INFO:teuthology.orchestra.run.smithi153.stdout:34359738375 2023-11-24T08:29:07.600 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738375 for osd.0 2023-11-24T08:29:07.601 DEBUG:teuthology.parallel:result is None 2023-11-24T08:29:07.601 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-11-24T08:29:07.601 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-24T08:29:07.918 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:07.919 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-24T08:29:07.934 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-11-24T08:29:06.585061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.789022"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.825380+0000","last_change":"2023-11-24T08:29:04.820862+0000","last_active":"2023-11-24T08:29:04.825380+0000","last_peered":"2023-11-24T08:29:04.825380+0000","last_clean":"2023-11-24T08:29:04.825380+0000","last_became_active":"2023-11-24T08:29:04.818386+0000","last_became_peered":"2023-11-24T08:29:04.818386+0000","last_unstale":"2023-11-24T08:29:04.825380+0000","last_undegraded":"2023-11-24T08:29:04.825380+0000","last_fullsized":"2023-11-24T08:29:04.825380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:42.310199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.823023+0000","last_change":"2023-11-24T08:29:04.818405+0000","last_active":"2023-11-24T08:29:04.823023+0000","last_peered":"2023-11-24T08:29:04.823023+0000","last_clean":"2023-11-24T08:29:04.823023+0000","last_became_active":"2023-11-24T08:29:04.817967+0000","last_became_peered":"2023-11-24T08:29:04.817967+0000","last_unstale":"2023-11-24T08:29:04.823023+0000","last_undegraded":"2023-11-24T08:29:04.823023+0000","last_fullsized":"2023-11-24T08:29:04.823023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:11:07.101911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.825312+0000","last_change":"2023-11-24T08:29:04.819596+0000","last_active":"2023-11-24T08:29:04.825312+0000","last_peered":"2023-11-24T08:29:04.825312+0000","last_clean":"2023-11-24T08:29:04.825312+0000","last_became_active":"2023-11-24T08:29:04.818295+0000","last_became_peered":"2023-11-24T08:29:04.818295+0000","last_unstale":"2023-11-24T08:29:04.825312+0000","last_undegraded":"2023-11-24T08:29:04.825312+0000","last_fullsized":"2023-11-24T08:29:04.825312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:58:43.217235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.825342+0000","last_change":"2023-11-24T08:29:04.820593+0000","last_active":"2023-11-24T08:29:04.825342+0000","last_peered":"2023-11-24T08:29:04.825342+0000","last_clean":"2023-11-24T08:29:04.825342+0000","last_became_active":"2023-11-24T08:29:04.818750+0000","last_became_peered":"2023-11-24T08:29:04.818750+0000","last_unstale":"2023-11-24T08:29:04.825342+0000","last_undegraded":"2023-11-24T08:29:04.825342+0000","last_fullsized":"2023-11-24T08:29:04.825342+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:12.615810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.821998+0000","last_change":"2023-11-24T08:29:04.820326+0000","last_active":"2023-11-24T08:29:04.821998+0000","last_peered":"2023-11-24T08:29:04.821998+0000","last_clean":"2023-11-24T08:29:04.821998+0000","last_became_active":"2023-11-24T08:29:04.819285+0000","last_became_peered":"2023-11-24T08:29:04.819285+0000","last_unstale":"2023-11-24T08:29:04.821998+0000","last_undegraded":"2023-11-24T08:29:04.821998+0000","last_fullsized":"2023-11-24T08:29:04.821998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:10:14.045668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-24T08:29:05.810106+0000","last_change":"2023-11-24T08:29:04.819744+0000","last_active":"2023-11-24T08:29:05.810106+0000","last_peered":"2023-11-24T08:29:05.810106+0000","last_clean":"2023-11-24T08:29:05.810106+0000","last_became_active":"2023-11-24T08:29:04.819191+0000","last_became_peered":"2023-11-24T08:29:04.819191+0000","last_unstale":"2023-11-24T08:29:05.810106+0000","last_undegraded":"2023-11-24T08:29:05.810106+0000","last_fullsized":"2023-11-24T08:29:05.810106+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:09:02.746576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-24T08:29:05.810105+0000","last_change":"2023-11-24T08:29:04.821790+0000","last_active":"2023-11-24T08:29:05.810105+0000","last_peered":"2023-11-24T08:29:05.810105+0000","last_clean":"2023-11-24T08:29:05.810105+0000","last_became_active":"2023-11-24T08:29:04.821369+0000","last_became_peered":"2023-11-24T08:29:04.821369+0000","last_unstale":"2023-11-24T08:29:05.810105+0000","last_undegraded":"2023-11-24T08:29:05.810105+0000","last_fullsized":"2023-11-24T08:29:05.810105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:52:15.752249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.825383+0000","last_change":"2023-11-24T08:29:04.820777+0000","last_active":"2023-11-24T08:29:04.825383+0000","last_peered":"2023-11-24T08:29:04.825383+0000","last_clean":"2023-11-24T08:29:04.825383+0000","last_became_active":"2023-11-24T08:29:04.817968+0000","last_became_peered":"2023-11-24T08:29:04.817968+0000","last_unstale":"2023-11-24T08:29:04.825383+0000","last_undegraded":"2023-11-24T08:29:04.825383+0000","last_fullsized":"2023-11-24T08:29:04.825383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:09:18.509147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-24T08:29:03.798542+0000","last_change":"2023-11-24T08:28:59.784048+0000","last_active":"2023-11-24T08:29:03.798542+0000","last_peered":"2023-11-24T08:29:03.798542+0000","last_clean":"2023-11-24T08:29:03.798542+0000","last_became_active":"2023-11-24T08:28:59.783498+0000","last_became_peered":"2023-11-24T08:28:59.783498+0000","last_unstale":"2023-11-24T08:29:03.798542+0000","last_undegraded":"2023-11-24T08:29:03.798542+0000","last_fullsized":"2023-11-24T08:29:03.798542+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_clean_scrub_stamp":"2023-11-24T08:28:58.760510+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:12:39.581115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:29:07.936 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-24T08:29:08.193 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:08.193 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-24T08:29:08.209 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-11-24T08:29:06.585061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82556,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032964,"statfs":{"total":289910292480,"available":289825755136,"internally_reserved":0,"allocated":1531904,"data_stored":1274729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.789022"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.825380+0000","last_change":"2023-11-24T08:29:04.820862+0000","last_active":"2023-11-24T08:29:04.825380+0000","last_peered":"2023-11-24T08:29:04.825380+0000","last_clean":"2023-11-24T08:29:04.825380+0000","last_became_active":"2023-11-24T08:29:04.818386+0000","last_became_peered":"2023-11-24T08:29:04.818386+0000","last_unstale":"2023-11-24T08:29:04.825380+0000","last_undegraded":"2023-11-24T08:29:04.825380+0000","last_fullsized":"2023-11-24T08:29:04.825380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:39:42.310199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.823023+0000","last_change":"2023-11-24T08:29:04.818405+0000","last_active":"2023-11-24T08:29:04.823023+0000","last_peered":"2023-11-24T08:29:04.823023+0000","last_clean":"2023-11-24T08:29:04.823023+0000","last_became_active":"2023-11-24T08:29:04.817967+0000","last_became_peered":"2023-11-24T08:29:04.817967+0000","last_unstale":"2023-11-24T08:29:04.823023+0000","last_undegraded":"2023-11-24T08:29:04.823023+0000","last_fullsized":"2023-11-24T08:29:04.823023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:11:07.101911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.825312+0000","last_change":"2023-11-24T08:29:04.819596+0000","last_active":"2023-11-24T08:29:04.825312+0000","last_peered":"2023-11-24T08:29:04.825312+0000","last_clean":"2023-11-24T08:29:04.825312+0000","last_became_active":"2023-11-24T08:29:04.818295+0000","last_became_peered":"2023-11-24T08:29:04.818295+0000","last_unstale":"2023-11-24T08:29:04.825312+0000","last_undegraded":"2023-11-24T08:29:04.825312+0000","last_fullsized":"2023-11-24T08:29:04.825312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:58:43.217235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.825342+0000","last_change":"2023-11-24T08:29:04.820593+0000","last_active":"2023-11-24T08:29:04.825342+0000","last_peered":"2023-11-24T08:29:04.825342+0000","last_clean":"2023-11-24T08:29:04.825342+0000","last_became_active":"2023-11-24T08:29:04.818750+0000","last_became_peered":"2023-11-24T08:29:04.818750+0000","last_unstale":"2023-11-24T08:29:04.825342+0000","last_undegraded":"2023-11-24T08:29:04.825342+0000","last_fullsized":"2023-11-24T08:29:04.825342+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:48:12.615810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.821998+0000","last_change":"2023-11-24T08:29:04.820326+0000","last_active":"2023-11-24T08:29:04.821998+0000","last_peered":"2023-11-24T08:29:04.821998+0000","last_clean":"2023-11-24T08:29:04.821998+0000","last_became_active":"2023-11-24T08:29:04.819285+0000","last_became_peered":"2023-11-24T08:29:04.819285+0000","last_unstale":"2023-11-24T08:29:04.821998+0000","last_undegraded":"2023-11-24T08:29:04.821998+0000","last_fullsized":"2023-11-24T08:29:04.821998+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:10:14.045668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-24T08:29:05.810106+0000","last_change":"2023-11-24T08:29:04.819744+0000","last_active":"2023-11-24T08:29:05.810106+0000","last_peered":"2023-11-24T08:29:05.810106+0000","last_clean":"2023-11-24T08:29:05.810106+0000","last_became_active":"2023-11-24T08:29:04.819191+0000","last_became_peered":"2023-11-24T08:29:04.819191+0000","last_unstale":"2023-11-24T08:29:05.810106+0000","last_undegraded":"2023-11-24T08:29:05.810106+0000","last_fullsized":"2023-11-24T08:29:05.810106+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:09:02.746576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-24T08:29:05.810105+0000","last_change":"2023-11-24T08:29:04.821790+0000","last_active":"2023-11-24T08:29:05.810105+0000","last_peered":"2023-11-24T08:29:05.810105+0000","last_clean":"2023-11-24T08:29:05.810105+0000","last_became_active":"2023-11-24T08:29:04.821369+0000","last_became_peered":"2023-11-24T08:29:04.821369+0000","last_unstale":"2023-11-24T08:29:05.810105+0000","last_undegraded":"2023-11-24T08:29:05.810105+0000","last_fullsized":"2023-11-24T08:29:05.810105+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:52:15.752249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-11-24T08:29:04.825383+0000","last_change":"2023-11-24T08:29:04.820777+0000","last_active":"2023-11-24T08:29:04.825383+0000","last_peered":"2023-11-24T08:29:04.825383+0000","last_clean":"2023-11-24T08:29:04.825383+0000","last_became_active":"2023-11-24T08:29:04.817968+0000","last_became_peered":"2023-11-24T08:29:04.817968+0000","last_unstale":"2023-11-24T08:29:04.825383+0000","last_undegraded":"2023-11-24T08:29:04.825383+0000","last_fullsized":"2023-11-24T08:29:04.825383+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:09:18.509147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-24T08:29:03.798542+0000","last_change":"2023-11-24T08:28:59.784048+0000","last_active":"2023-11-24T08:29:03.798542+0000","last_peered":"2023-11-24T08:29:03.798542+0000","last_clean":"2023-11-24T08:29:03.798542+0000","last_became_active":"2023-11-24T08:28:59.783498+0000","last_became_peered":"2023-11-24T08:28:59.783498+0000","last_unstale":"2023-11-24T08:29:03.798542+0000","last_undegraded":"2023-11-24T08:29:03.798542+0000","last_fullsized":"2023-11-24T08:29:03.798542+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_clean_scrub_stamp":"2023-11-24T08:28:58.760510+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:12:39.581115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27132,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344708,"statfs":{"total":96636764160,"available":96608980992,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:29:08.210 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-11-24T08:29:08.210 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-11-24T08:29:08.210 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-11-24T08:29:08.211 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-11-24T08:29:08.500 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:08.501 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-24T08:29:08.518 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-11-24T08:29:08.519 INFO:teuthology.run_tasks:Running task ceph... 2023-11-24T08:29:08.527 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-11-24T08:29:08.528 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-24T08:29:08.528 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7465501', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2661, 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1', 'sleep_before_teardown': 0, 'subset': '14/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1', 'targets': {'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKay/u69gbSDH+wQg1rANAqFcLSHi3oa5yrvt7QTzuCp6Tnk1XI0S9vKeGRUUVuR7yn7Ern95TqUza/7zUGQSdo=', 'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNN4flvWtU6IA2MNQJD2lLFhHIgd/PZ0iehpJcUsE7kQw/8ZZUYwpfu5sODRLPlVm8TccYiZoQyDcPZAMHRMrk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '2442239d2653456406c25ae0c71b689c8f2657b6', 'timestamp': '2023-11-24_05:31:39', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-24T08:29:08.528 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-11-24T08:29:08.530 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-11-24T08:29:08.537 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T08:29:08.579 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T08:29:08.593 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:08.593 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2023-11-24T08:29:08.646 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-24T08:29:08.646 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2023-11-24T08:29:08.694 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-24T08:29:08.694 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:29:08.694 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 888 Links: 1 2023-11-24T08:29:08.694 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:29:08.694 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-11-24 08:23:11.424946064 +0000 2023-11-24T08:29:08.694 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-11-24 08:23:11.068949244 +0000 2023-11-24T08:29:08.694 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-11-24 08:23:11.068949244 +0000 2023-11-24T08:29:08.695 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2023-11-24T08:29:08.695 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-24T08:29:08.748 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-11-24T08:29:08.748 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-11-24T08:29:08.749 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00039022 s, 1.3 MB/s 2023-11-24T08:29:08.749 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-24T08:29:08.799 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2023-11-24T08:29:08.846 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-24T08:29:08.846 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:29:08.846 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 909 Links: 1 2023-11-24T08:29:08.846 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:29:08.846 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-11-24 08:23:11.988941025 +0000 2023-11-24T08:29:08.846 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-11-24 08:23:11.660943956 +0000 2023-11-24T08:29:08.846 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-11-24 08:23:11.660943956 +0000 2023-11-24T08:29:08.847 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2023-11-24T08:29:08.847 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-24T08:29:08.900 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-11-24T08:29:08.901 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-11-24T08:29:08.901 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000418414 s, 1.2 MB/s 2023-11-24T08:29:08.901 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-24T08:29:08.951 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2023-11-24T08:29:08.998 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-24T08:29:08.998 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:29:08.998 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 924 Links: 1 2023-11-24T08:29:08.998 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:29:08.998 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-11-24 08:23:12.232938847 +0000 2023-11-24T08:29:08.998 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-11-24 08:23:12.228938883 +0000 2023-11-24T08:29:08.998 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-11-24 08:23:12.228938883 +0000 2023-11-24T08:29:08.998 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2023-11-24T08:29:08.999 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-24T08:29:09.052 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-11-24T08:29:09.052 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-11-24T08:29:09.052 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000375178 s, 1.4 MB/s 2023-11-24T08:29:09.054 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-24T08:29:09.102 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2023-11-24T08:29:09.154 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-24T08:29:09.154 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T08:29:09.154 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 943 Links: 1 2023-11-24T08:29:09.154 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T08:29:09.154 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2023-11-24 08:23:13.164930522 +0000 2023-11-24T08:29:09.154 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2023-11-24 08:23:12.832933488 +0000 2023-11-24T08:29:09.154 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2023-11-24 08:23:12.832933488 +0000 2023-11-24T08:29:09.154 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2023-11-24T08:29:09.155 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-24T08:29:09.208 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2023-11-24T08:29:09.208 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2023-11-24T08:29:09.208 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000375668 s, 1.4 MB/s 2023-11-24T08:29:09.209 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-24T08:29:09.259 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-24T08:29:09.259 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-24T08:29:09.259 INFO:tasks.ceph:Generating config... 2023-11-24T08:29:09.260 INFO:tasks.ceph:[client] debug rbd = 20 2023-11-24T08:29:09.260 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-11-24T08:29:09.260 INFO:tasks.ceph:[client] log to stderr = False 2023-11-24T08:29:09.260 INFO:tasks.ceph:[client] rbd default features = 125 2023-11-24T08:29:09.261 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-11-24T08:29:09.261 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.261 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.261 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.261 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.261 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.261 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.261 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.261 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.261 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-11-24T08:29:09.261 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-11-24T08:29:09.261 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-11-24T08:29:09.262 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-24T08:29:09.262 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-24T08:29:09.262 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-24T08:29:09.262 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-24T08:29:09.262 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-24T08:29:09.262 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-24T08:29:09.262 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-24T08:29:09.263 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-24T08:29:09.264 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-24T08:29:09.264 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-11-24T08:29:09.264 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-11-24T08:29:09.332 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.keyring 2023-11-24T08:29:09.334 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-11-24T08:29:09.413 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-11-24T08:29:09.468 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.164')] 2023-11-24T08:29:09.469 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.164', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-11-24T08:29:09.470 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:09.470 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T08:29:09.517 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.164 --print /home/ubuntu/cephtest/cluster2.monmap 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: generated fsid 6f0e426c-18ac-4eb5-9ad7-dc680622a77c 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:setting min_mon_release = pacific 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:epoch 0 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:fsid 6f0e426c-18ac-4eb5-9ad7-dc680622a77c 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:last_changed 2023-11-24T08:29:09.598743+0000 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:created 2023-11-24T08:29:09.598743+0000 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:min_mon_release 16 (pacific) 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:election_strategy: 1 2023-11-24T08:29:09.597 INFO:teuthology.orchestra.run.smithi164.stdout:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2023-11-24T08:29:09.598 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-11-24T08:29:09.599 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T08:29:09.645 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 6f0e426c-18ac-4eb5-9ad7-dc680622a77c... 2023-11-24T08:29:09.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-11-24T08:29:09.649 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-11-24T08:29:09.696 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2023-11-24T08:29:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2023-11-24T08:29:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T08:29:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2023-11-24T08:29:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2023-11-24T08:29:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2023-11-24T08:29:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2023-11-24T08:29:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2023-11-24T08:29:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2023-11-24T08:29:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2023-11-24T08:29:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T08:29:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2023-11-24T08:29:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T08:29:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T08:29:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2023-11-24T08:29:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2023-11-24T08:29:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2023-11-24T08:29:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2023-11-24T08:29:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2023-11-24T08:29:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2023-11-24T08:29:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2023-11-24T08:29:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2023-11-24T08:29:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T08:29:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T08:29:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2023-11-24T08:29:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T08:29:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2023-11-24T08:29:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2023-11-24T08:29:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2023-11-24T08:29:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2023-11-24T08:29:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2023-11-24T08:29:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = 172.21.15.164 2023-11-24T08:29:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2023-11-24T08:29:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on pool no app = False 2023-11-24T08:29:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 5000 2023-11-24T08:29:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 6f0e426c-18ac-4eb5-9ad7-dc680622a77c 2023-11-24T08:29:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.703 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2023-11-24T08:29:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2023-11-24T08:29:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2023-11-24T08:29:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2023-11-24T08:29:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2023-11-24T08:29:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2023-11-24T08:29:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2023-11-24T08:29:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2023-11-24T08:29:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2023-11-24T08:29:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2023-11-24T08:29:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2023-11-24T08:29:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2023-11-24T08:29:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2023-11-24T08:29:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: bdev async discard = True 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: bdev enable discard = True 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore allocator = stupid 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 1/20 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 1/20 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 4/10 2023-11-24T08:29:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:29:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2023-11-24T08:29:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2023-11-24T08:29:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: debug rbd = 20 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: debug rbd_mirror = 15 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: log to stderr = False 2023-11-24T08:29:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: rbd default features = 125 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: rbd mirror image policy type = simple 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.0] 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.1] 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.2] 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout:[client.mirror.3] 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.722 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2023-11-24T08:29:09.722 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2023-11-24T08:29:09.722 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2023-11-24T08:29:09.723 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2023-11-24T08:29:09.724 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2023-11-24T08:29:09.725 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2023-11-24T08:29:09.725 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2023-11-24T08:29:09.725 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T08:29:09.725 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T08:29:09.725 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2023-11-24T08:29:09.725 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.725 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T08:29:09.725 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.725 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = 172.21.15.164 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: mon client directed command retry = 5 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on pool no app = False 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject socket failures = 5000 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = 6f0e426c-18ac-4eb5-9ad7-dc680622a77c 2023-11-24T08:29:09.726 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2023-11-24T08:29:09.727 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.728 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = stupid 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 1/20 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 1/20 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2023-11-24T08:29:09.729 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 4/10 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2023-11-24T08:29:09.730 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2023-11-24T08:29:09.731 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.732 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: debug rbd = 20 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: debug rbd_mirror = 15 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: log to stderr = False 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: rbd default features = 125 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: rbd mirror image policy type = simple 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2023-11-24T08:29:09.733 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.734 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.734 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2023-11-24T08:29:09.734 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.734 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.734 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2023-11-24T08:29:09.734 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.734 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.735 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2023-11-24T08:29:09.735 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-24T08:29:09.735 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-24T08:29:09.737 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-11-24T08:29:09.737 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-11-24T08:29:09.830 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-24T08:29:09.830 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:09.830 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-11-24T08:29:09.835 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:09.836 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-11-24T08:29:09.883 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T08:29:09.883 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:29:09.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/cluster2.keyring 2023-11-24T08:29:09.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-11-24T08:29:09.907 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:29:09.907 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-11-24T08:29:09.955 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T08:29:09.955 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:09.955 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/cluster2.keyring 2023-11-24T08:29:09.955 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-11-24T08:29:09.979 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:09.979 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-11-24T08:29:10.026 INFO:tasks.ceph:Setting up mon nodes... 2023-11-24T08:29:10.026 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-24T08:29:10.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-11-24T08:29:10.070 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-11-24T08:29:10.073 INFO:tasks.ceph:Setting up mds nodes... 2023-11-24T08:29:10.074 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-24T08:29:10.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-11-24T08:29:10.107 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-11-24T08:29:10.116 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-11-24T08:29:10.152 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-11-24T08:29:10.165 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-11-24T08:29:10.239 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-11-24T08:29:10.249 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-11-24T08:29:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-11-24T08:29:10.338 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-11-24T08:29:10.414 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-11-24T08:29:10.424 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-11-24T08:29:10.496 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-11-24T08:29:10.506 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-11-24T08:29:10.583 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-11-24T08:29:10.594 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-11-24T08:29:10.670 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-11-24T08:29:10.681 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-11-24T08:29:10.759 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-11-24T08:29:10.768 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-24T08:29:10.769 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-24T08:29:10.769 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-11-24T08:29:10.823 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-24T08:29:10.823 INFO:tasks.ceph:role: cluster2.osd.0 2023-11-24T08:29:10.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T08:29:10.823 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-24T08:29:10.885 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:29:10.885 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:29:10.885 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:29:10.885 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T08:29:10.885 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:29:10.885 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:29:10.885 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:29:10.885 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T08:29:10.886 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:29:10.886 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:29:11.366 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2023-11-24T08:29:11.368 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2023-11-24T08:29:11.369 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-11-24T08:29:11.489 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-11-24T08:29:11.501 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2023-11-24T08:29:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:29:11.502 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-11-24T08:29:11.557 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-24T08:29:11.557 INFO:tasks.ceph:role: cluster2.osd.1 2023-11-24T08:29:11.557 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T08:29:11.557 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-24T08:29:11.620 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:29:11.621 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:29:11.621 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:29:11.621 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T08:29:11.621 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:29:11.621 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:29:11.621 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:29:11.621 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T08:29:11.621 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:29:11.621 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:29:12.044 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2023-11-24T08:29:12.046 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2023-11-24T08:29:12.047 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-11-24T08:29:12.111 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-11-24T08:29:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:29:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2023-11-24T08:29:12.163 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-11-24T08:29:12.221 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-24T08:29:12.222 INFO:tasks.ceph:role: cluster2.osd.2 2023-11-24T08:29:12.222 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T08:29:12.222 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-24T08:29:12.290 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T08:29:12.290 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T08:29:12.290 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T08:29:12.290 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T08:29:12.290 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T08:29:12.291 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2023-11-24T08:29:12.291 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T08:29:12.291 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T08:29:12.291 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T08:29:12.291 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T08:29:12.752 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2023-11-24T08:29:12.755 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2023-11-24T08:29:12.756 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-11-24T08:29:12.849 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-11-24T08:29:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:29:12.905 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: /sbin/restorecon: command not found 2023-11-24T08:29:12.905 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-11-24T08:29:13.086 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:13.083+0000 7f8d129a8640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-11-24T08:29:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:13.087+0000 7f8d129a8640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-11-24T08:29:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:13.087+0000 7f8d129a8640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-11-24T08:29:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:13.087+0000 7f8d129a8640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-11-24T08:29:13.092 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:13.087+0000 7f8d129a8640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-11-24T08:29:13.093 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:13.091+0000 7f8d129a8640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-11-24T08:29:16.104 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-11-24T08:29:16.118 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-11-24T08:29:16.205 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:16.203+0000 7f59339d9640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-11-24T08:29:16.206 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:16.203+0000 7f59339d9640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-11-24T08:29:16.206 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:16.203+0000 7f59339d9640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-11-24T08:29:16.206 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:16.203+0000 7f59339d9640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-11-24T08:29:16.206 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:16.203+0000 7f59339d9640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-11-24T08:29:16.206 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:16.203+0000 7f59339d9640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-11-24T08:29:19.210 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-11-24T08:29:19.266 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-11-24T08:29:19.349 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:19.343+0000 7f8b2a58f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-11-24T08:29:19.350 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:19.347+0000 7f8b2a58f640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-11-24T08:29:19.350 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:19.347+0000 7f8b2a58f640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-11-24T08:29:19.350 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:19.347+0000 7f8b2a58f640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-11-24T08:29:19.350 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:19.347+0000 7f8b2a58f640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-11-24T08:29:19.350 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T08:29:19.347+0000 7f8b2a58f640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-11-24T08:29:22.349 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-11-24T08:29:22.403 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-24T08:29:22.403 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:29:22.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-11-24T08:29:22.418 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.418 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-11-24T08:29:22.456 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.456 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-11-24T08:29:22.510 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.510 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-11-24T08:29:22.564 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-24T08:29:22.564 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-11-24T08:29:22.570 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.570 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-11-24T08:29:22.610 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.610 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-11-24T08:29:22.657 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.658 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-11-24T08:29:22.706 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.706 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-11-24T08:29:22.754 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.754 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-11-24T08:29:22.802 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.802 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-11-24T08:29:22.850 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.850 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-11-24T08:29:22.898 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:22.898 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-11-24T08:29:22.946 INFO:tasks.ceph:Adding keys to all mons... 2023-11-24T08:29:22.946 DEBUG:teuthology.orchestra.run.smithi164:> sudo tee -a /etc/ceph/cluster2.keyring 2023-11-24T08:29:23.000 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2023-11-24T08:29:23.000 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlgP5SBBAAz3nU+ERcU5O149fsOmJPaw== 2023-11-24T08:29:23.000 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2023-11-24T08:29:23.000 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDZXmBlxdlEBRAAWo3VdxMAuZsIrSmHD632qQ== 2023-11-24T08:29:23.000 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2023-11-24T08:29:23.000 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDcXmBlphNZDBAALXBHNSj/S47OMQqYXzUvGw== 2023-11-24T08:29:23.000 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDfXmBlOivpFBAAcT40qtpWtLxr7/OJtlUP1A== 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBl+qd/BhAA0QEsBl9o1CRkNq4i02FU+Q== 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror] 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBliYkpCRAAXzQ6jLSavAPjhXjZRYC66A== 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlModdDhAA4+Qogd32KBPOVJtOT7ZTHg== 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlqqybExAAVMrLMjuC+Y/9fwO+IWkLTQ== 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlFUTLGBAAp8z3EwxyCaLDRcLW2dXkYg== 2023-11-24T08:29:23.001 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2023-11-24T08:29:23.002 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlq1ehHRAAL56tB63zg/r/YSWQDCnMoA== 2023-11-24T08:29:23.002 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.4] 2023-11-24T08:29:23.002 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlYQTWIhAAOsG3BSexy1oNnSOa/Xjnkg== 2023-11-24T08:29:23.002 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.5] 2023-11-24T08:29:23.002 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlUaUQKBAA+kSdcb+J0RtZicom+VDLWQ== 2023-11-24T08:29:23.002 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.6] 2023-11-24T08:29:23.002 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBl74VZLRAAaj6Y8U2yC0dnOnyhzsqfVw== 2023-11-24T08:29:23.003 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T08:29:23.089 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:29:23.130 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:29:23.212 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T08:29:23.293 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:29:23.379 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:29:23.467 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:29:23.553 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:29:23.641 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:29:23.726 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:29:23.811 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:29:23.895 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:29:23.979 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T08:29:24.063 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-24T08:29:24.064 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-11-24T08:29:24.116 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-11-24T08:29:24.209 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-11-24T08:29:24.219 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-11-24T08:29:24.265 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-11-24T08:29:24.265 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-11-24T08:29:24.266 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-11-24T08:29:24.310 INFO:tasks.ceph.cluster2.mon.a:Started 2023-11-24T08:29:24.310 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-11-24T08:29:24.313 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-11-24T08:29:24.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-11-24T08:29:24.316 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-11-24T08:29:24.316 INFO:tasks.ceph:Setting crush tunables to default 2023-11-24T08:29:24.316 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd crush tunables default 2023-11-24T08:29:24.558 INFO:teuthology.orchestra.run.smithi164.stderr:adjusted tunables profile to default 2023-11-24T08:29:24.575 INFO:tasks.ceph:check_enable_crimson: False 2023-11-24T08:29:24.575 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-11-24T08:29:24.575 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:24.575 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-11-24T08:29:24.585 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:24.585 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-11-24T08:29:24.640 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:29:24.640 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-11-24T08:29:24.692 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd new 6258618b-d8dc-44df-ab90-e9449ba427cb 0 2023-11-24T08:29:24.719 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:24.716+0000 7fd965a50180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-24T08:29:24.944 INFO:teuthology.orchestra.run.smithi164.stdout:0 2023-11-24T08:29:24.961 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd new ca72c49f-fcd7-47ad-9ee0-429a15a9e25e 1 2023-11-24T08:29:25.118 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:25.116+0000 7fd965a50180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-24T08:29:25.163 INFO:teuthology.orchestra.run.smithi164.stdout:1 2023-11-24T08:29:25.180 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd new 40a2ccc1-18ff-42eb-8094-6a7343b4d6e1 2 2023-11-24T08:29:25.259 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:25.256+0000 7fd965a50180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-24T08:29:25.373 INFO:teuthology.orchestra.run.smithi164.stdout:2 2023-11-24T08:29:25.390 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-11-24T08:29:25.390 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-11-24T08:29:25.392 INFO:tasks.ceph.cluster2.osd.0:Started 2023-11-24T08:29:25.392 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-11-24T08:29:25.393 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-11-24T08:29:25.395 INFO:tasks.ceph.cluster2.osd.1:Started 2023-11-24T08:29:25.395 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-11-24T08:29:25.395 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-11-24T08:29:25.397 INFO:tasks.ceph.cluster2.osd.2:Started 2023-11-24T08:29:25.397 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-11-24T08:29:25.575 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:25.576 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":5,"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","created":"2023-11-24T08:29:24.450546+0000","modified":"2023-11-24T08:29:25.373302+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T08:29:25.373302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6258618b-d8dc-44df-ab90-e9449ba427cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ca72c49f-fcd7-47ad-9ee0-429a15a9e25e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"40a2ccc1-18ff-42eb-8094-6a7343b4d6e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:29:25.592 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-11-24T08:29:25.593 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-24T08:29:25.684 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:25.680+0000 7fd965a50180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-24T08:29:25.734 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:25.732+0000 7fd965a50180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-24T08:29:25.783 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:25.780+0000 7fd965a50180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-24T08:29:25.856 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:25.852+0000 7fd965a50180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-24T08:29:25.895 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-11-24T08:29:25.904 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:25.900+0000 7fd965a50180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-24T08:29:25.977 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:25.972+0000 7fd965a50180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-24T08:29:26.015 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:29:26.011+0000 7f42c1c0a640 -1 Falling back to public interface 2023-11-24T08:29:26.025 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:26.020+0000 7fd965a50180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-24T08:29:26.026 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T08:29:26.023+0000 7fee40cc2640 -1 Falling back to public interface 2023-11-24T08:29:26.034 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T08:29:26.031+0000 7fcebbba7640 -1 Falling back to public interface 2023-11-24T08:29:26.091 INFO:teuthology.misc.health.smithi164.stdout: 2023-11-24T08:29:26.091 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":5,"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","created":"2023-11-24T08:29:24.450546+0000","modified":"2023-11-24T08:29:25.373302+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T08:29:25.373302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6258618b-d8dc-44df-ab90-e9449ba427cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ca72c49f-fcd7-47ad-9ee0-429a15a9e25e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"40a2ccc1-18ff-42eb-8094-6a7343b4d6e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:29:26.107 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-24T08:29:26.153 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:26.148+0000 7fd965a50180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-24T08:29:26.203 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:26.200+0000 7fd965a50180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-24T08:29:26.420 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-11-24T08:29:26.420 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-11-24T08:29:26.420 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr: from numpy import show_config as show_numpy_config 2023-11-24T08:29:26.427 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:26.424+0000 7fd965a50180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-24T08:29:26.548 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:26.544+0000 7fd965a50180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-24T08:29:26.656 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:26.652+0000 7fd965a50180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-24T08:29:26.717 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:26.712+0000 7fd965a50180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-24T08:29:26.838 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:26.836+0000 7fd965a50180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-24T08:29:27.308 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:27.304+0000 7fd965a50180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-24T08:29:27.379 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:27.376+0000 7fd965a50180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-24T08:29:27.512 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:27.508+0000 7fd965a50180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-24T08:29:27.583 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:27.580+0000 7fd965a50180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-24T08:29:27.861 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:27.856+0000 7fd965a50180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-24T08:29:27.913 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:27.908+0000 7fd965a50180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-24T08:29:28.949 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T08:29:28.944+0000 7fd90d2d9640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-24T08:29:29.810 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T08:29:29.807+0000 7fee40cc2640 -1 osd.1 0 log_to_monitors true 2023-11-24T08:29:29.827 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:29:29.823+0000 7f42c1c0a640 -1 osd.2 0 log_to_monitors true 2023-11-24T08:29:29.858 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T08:29:29.855+0000 7fcebbba7640 -1 osd.0 0 log_to_monitors true 2023-11-24T08:29:32.410 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-11-24T08:29:32.849 INFO:teuthology.misc.health.smithi164.stdout: 2023-11-24T08:29:32.849 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":7,"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","created":"2023-11-24T08:29:24.450546+0000","modified":"2023-11-24T08:29:31.404415+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T08:29:25.373302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6258618b-d8dc-44df-ab90-e9449ba427cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ca72c49f-fcd7-47ad-9ee0-429a15a9e25e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"40a2ccc1-18ff-42eb-8094-6a7343b4d6e1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:29:32.874 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-24T08:29:33.339 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T08:29:33.335+0000 7fee3c449640 -1 osd.1 0 waiting for initial osdmap 2023-11-24T08:29:33.374 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T08:29:33.371+0000 7fee34a96640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:29:33.397 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T08:29:33.395+0000 7fceb8b64640 -1 osd.0 0 waiting for initial osdmap 2023-11-24T08:29:33.397 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:29:33.395+0000 7f42bebc7640 -1 osd.2 0 waiting for initial osdmap 2023-11-24T08:29:33.430 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T08:29:33.427+0000 7fceaf97b640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:29:33.430 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:29:33.427+0000 7f42b59de640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T08:29:39.177 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-11-24T08:29:39.475 INFO:teuthology.misc.health.smithi164.stdout: 2023-11-24T08:29:39.475 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":12,"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","created":"2023-11-24T08:29:24.450546+0000","modified":"2023-11-24T08:29:38.513739+0000","last_up_change":"2023-11-24T08:29:34.343008+0000","last_in_change":"2023-11-24T08:29:25.373302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:29:36.023340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6258618b-d8dc-44df-ab90-e9449ba427cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6817","nonce":2800571832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6819","nonce":2800571832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6823","nonce":2800571832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6821","nonce":2800571832}]},"public_addr":"172.21.15.164:6817/2800571832","cluster_addr":"172.21.15.164:6819/2800571832","heartbeat_back_addr":"172.21.15.164:6823/2800571832","heartbeat_front_addr":"172.21.15.164:6821/2800571832","state":["exists","up"]},{"osd":1,"uuid":"ca72c49f-fcd7-47ad-9ee0-429a15a9e25e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6809","nonce":3805995825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6811","nonce":3805995825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6815","nonce":3805995825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6813","nonce":3805995825}]},"public_addr":"172.21.15.164:6809/3805995825","cluster_addr":"172.21.15.164:6811/3805995825","heartbeat_back_addr":"172.21.15.164:6815/3805995825","heartbeat_front_addr":"172.21.15.164:6813/3805995825","state":["exists","up"]},{"osd":2,"uuid":"40a2ccc1-18ff-42eb-8094-6a7343b4d6e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6801","nonce":2943801369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6803","nonce":2943801369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6807","nonce":2943801369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6805","nonce":2943801369}]},"public_addr":"172.21.15.164:6801/2943801369","cluster_addr":"172.21.15.164:6803/2943801369","heartbeat_back_addr":"172.21.15.164:6807/2943801369","heartbeat_front_addr":"172.21.15.164:6805/2943801369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:29:30.902796+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:29:30.834150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:29:30.872220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:29:39.492 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-24T08:29:39.493 INFO:tasks.ceph:Creating RBD pool 2023-11-24T08:29:39.493 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-11-24T08:29:40.524 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'rbd' created 2023-11-24T08:29:40.552 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-24T08:29:41.526 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-24T08:29:41.556 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-11-24T08:29:41.557 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-11-24T08:29:41.557 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-11-24T08:29:41.557 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-11-24T08:29:41.557 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-24T08:29:41.880 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:41.897 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3888124239},{"type":"v1","addr":"172.21.15.153:6827","nonce":3888124239}]},"active_addr":"172.21.15.153:6827/3888124239","active_change":"2023-11-24T08:29:27.934296+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1024589526}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1799867781}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":629623340}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":2948720878}]}]} 2023-11-24T08:29:41.899 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-11-24T08:29:41.899 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-11-24T08:29:41.900 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-11-24T08:29:42.190 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:42.190 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":14,"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","created":"2023-11-24T08:29:24.450546+0000","modified":"2023-11-24T08:29:41.525637+0000","last_up_change":"2023-11-24T08:29:34.343008+0000","last_in_change":"2023-11-24T08:29:25.373302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:29:36.023340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-24T08:29:39.824470+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6258618b-d8dc-44df-ab90-e9449ba427cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6817","nonce":2800571832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6819","nonce":2800571832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6823","nonce":2800571832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6821","nonce":2800571832}]},"public_addr":"172.21.15.164:6817/2800571832","cluster_addr":"172.21.15.164:6819/2800571832","heartbeat_back_addr":"172.21.15.164:6823/2800571832","heartbeat_front_addr":"172.21.15.164:6821/2800571832","state":["exists","up"]},{"osd":1,"uuid":"ca72c49f-fcd7-47ad-9ee0-429a15a9e25e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6809","nonce":3805995825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6811","nonce":3805995825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6815","nonce":3805995825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6813","nonce":3805995825}]},"public_addr":"172.21.15.164:6809/3805995825","cluster_addr":"172.21.15.164:6811/3805995825","heartbeat_back_addr":"172.21.15.164:6815/3805995825","heartbeat_front_addr":"172.21.15.164:6813/3805995825","state":["exists","up"]},{"osd":2,"uuid":"40a2ccc1-18ff-42eb-8094-6a7343b4d6e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6801","nonce":2943801369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6803","nonce":2943801369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6807","nonce":2943801369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6805","nonce":2943801369}]},"public_addr":"172.21.15.164:6801/2943801369","cluster_addr":"172.21.15.164:6803/2943801369","heartbeat_back_addr":"172.21.15.164:6807/2943801369","heartbeat_front_addr":"172.21.15.164:6805/2943801369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:29:30.902796+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:29:30.834150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:29:30.872220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:29:42.208 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-11-24T08:29:42.208 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-11-24T08:29:42.493 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:42.493 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":14,"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","created":"2023-11-24T08:29:24.450546+0000","modified":"2023-11-24T08:29:41.525637+0000","last_up_change":"2023-11-24T08:29:34.343008+0000","last_in_change":"2023-11-24T08:29:25.373302+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:29:36.023340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-24T08:29:39.824470+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6258618b-d8dc-44df-ab90-e9449ba427cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6817","nonce":2800571832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6819","nonce":2800571832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6823","nonce":2800571832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":2800571832},{"type":"v1","addr":"172.21.15.164:6821","nonce":2800571832}]},"public_addr":"172.21.15.164:6817/2800571832","cluster_addr":"172.21.15.164:6819/2800571832","heartbeat_back_addr":"172.21.15.164:6823/2800571832","heartbeat_front_addr":"172.21.15.164:6821/2800571832","state":["exists","up"]},{"osd":1,"uuid":"ca72c49f-fcd7-47ad-9ee0-429a15a9e25e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6809","nonce":3805995825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6811","nonce":3805995825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6815","nonce":3805995825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3805995825},{"type":"v1","addr":"172.21.15.164:6813","nonce":3805995825}]},"public_addr":"172.21.15.164:6809/3805995825","cluster_addr":"172.21.15.164:6811/3805995825","heartbeat_back_addr":"172.21.15.164:6815/3805995825","heartbeat_front_addr":"172.21.15.164:6813/3805995825","state":["exists","up"]},{"osd":2,"uuid":"40a2ccc1-18ff-42eb-8094-6a7343b4d6e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6801","nonce":2943801369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6803","nonce":2943801369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6807","nonce":2943801369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":2943801369},{"type":"v1","addr":"172.21.15.164:6805","nonce":2943801369}]},"public_addr":"172.21.15.164:6801/2943801369","cluster_addr":"172.21.15.164:6803/2943801369","heartbeat_back_addr":"172.21.15.164:6807/2943801369","heartbeat_front_addr":"172.21.15.164:6805/2943801369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:29:30.902796+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:29:30.834150+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:29:30.872220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T08:29:42.510 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-11-24T08:29:42.511 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-11-24T08:29:42.511 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-11-24T08:29:42.674 INFO:teuthology.orchestra.run.smithi164.stdout:34359738374 2023-11-24T08:29:42.675 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-11-24T08:29:42.681 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2023-11-24T08:29:42.681 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-11-24T08:29:42.682 INFO:teuthology.orchestra.run.smithi164.stdout:34359738374 2023-11-24T08:29:42.683 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-11-24T08:29:42.983 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2023-11-24T08:29:43.000 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-11-24T08:29:43.016 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2023-11-24T08:29:43.031 INFO:teuthology.orchestra.run.smithi164.stdout:34359738372 2023-11-24T08:29:43.032 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-11-24T08:29:43.048 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2023-11-24T08:29:44.001 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-11-24T08:29:44.033 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-11-24T08:29:44.048 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-11-24T08:29:44.317 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2023-11-24T08:29:44.340 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2023-11-24T08:29:44.341 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-11-24T08:29:44.347 INFO:teuthology.orchestra.run.smithi164.stdout:34359738372 2023-11-24T08:29:44.358 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-11-24T08:29:44.365 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2023-11-24T08:29:45.342 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-11-24T08:29:45.359 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-11-24T08:29:45.365 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-11-24T08:29:45.662 INFO:teuthology.orchestra.run.smithi164.stdout:34359738374 2023-11-24T08:29:45.679 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2023-11-24T08:29:45.679 INFO:teuthology.orchestra.run.smithi164.stdout:34359738373 2023-11-24T08:29:45.680 DEBUG:teuthology.parallel:result is None 2023-11-24T08:29:45.689 INFO:teuthology.orchestra.run.smithi164.stdout:34359738374 2023-11-24T08:29:45.697 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.0 2023-11-24T08:29:45.697 DEBUG:teuthology.parallel:result is None 2023-11-24T08:29:45.707 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2023-11-24T08:29:45.707 DEBUG:teuthology.parallel:result is None 2023-11-24T08:29:45.707 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-11-24T08:29:45.707 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T08:29:46.011 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:46.011 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T08:29:46.027 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-11-24T08:29:45.946910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82852,"kb_used_data":1528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032668,"statfs":{"total":289910292480,"available":289825452032,"internally_reserved":0,"allocated":1564672,"data_stored":1285410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001558"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541670+0000","last_change":"2023-11-24T08:29:41.544251+0000","last_active":"2023-11-24T08:29:42.541670+0000","last_peered":"2023-11-24T08:29:42.541670+0000","last_clean":"2023-11-24T08:29:42.541670+0000","last_became_active":"2023-11-24T08:29:41.543720+0000","last_became_peered":"2023-11-24T08:29:41.543720+0000","last_unstale":"2023-11-24T08:29:42.541670+0000","last_undegraded":"2023-11-24T08:29:42.541670+0000","last_fullsized":"2023-11-24T08:29:42.541670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:40:21.348430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541291+0000","last_change":"2023-11-24T08:29:41.544774+0000","last_active":"2023-11-24T08:29:42.541291+0000","last_peered":"2023-11-24T08:29:42.541291+0000","last_clean":"2023-11-24T08:29:42.541291+0000","last_became_active":"2023-11-24T08:29:41.543928+0000","last_became_peered":"2023-11-24T08:29:41.543928+0000","last_unstale":"2023-11-24T08:29:42.541291+0000","last_undegraded":"2023-11-24T08:29:42.541291+0000","last_fullsized":"2023-11-24T08:29:42.541291+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:33.568413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541469+0000","last_change":"2023-11-24T08:29:41.544569+0000","last_active":"2023-11-24T08:29:42.541469+0000","last_peered":"2023-11-24T08:29:42.541469+0000","last_clean":"2023-11-24T08:29:42.541469+0000","last_became_active":"2023-11-24T08:29:41.543996+0000","last_became_peered":"2023-11-24T08:29:41.543996+0000","last_unstale":"2023-11-24T08:29:42.541469+0000","last_undegraded":"2023-11-24T08:29:42.541469+0000","last_fullsized":"2023-11-24T08:29:42.541469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:32:34.625958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541423+0000","last_change":"2023-11-24T08:29:41.546688+0000","last_active":"2023-11-24T08:29:42.541423+0000","last_peered":"2023-11-24T08:29:42.541423+0000","last_clean":"2023-11-24T08:29:42.541423+0000","last_became_active":"2023-11-24T08:29:41.546024+0000","last_became_peered":"2023-11-24T08:29:41.546024+0000","last_unstale":"2023-11-24T08:29:42.541423+0000","last_undegraded":"2023-11-24T08:29:42.541423+0000","last_fullsized":"2023-11-24T08:29:42.541423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:49:39.572701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:43.918845+0000","last_change":"2023-11-24T08:29:41.548070+0000","last_active":"2023-11-24T08:29:43.918845+0000","last_peered":"2023-11-24T08:29:43.918845+0000","last_clean":"2023-11-24T08:29:43.918845+0000","last_became_active":"2023-11-24T08:29:41.547695+0000","last_became_peered":"2023-11-24T08:29:41.547695+0000","last_unstale":"2023-11-24T08:29:43.918845+0000","last_undegraded":"2023-11-24T08:29:43.918845+0000","last_fullsized":"2023-11-24T08:29:43.918845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:30:28.663728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:43.918960+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T08:29:43.918960+0000","last_peered":"2023-11-24T08:29:43.918960+0000","last_clean":"2023-11-24T08:29:43.918960+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T08:29:43.918960+0000","last_undegraded":"2023-11-24T08:29:43.918960+0000","last_fullsized":"2023-11-24T08:29:43.918960+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:43.918857+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T08:29:43.918857+0000","last_peered":"2023-11-24T08:29:43.918857+0000","last_clean":"2023-11-24T08:29:43.918857+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T08:29:43.918857+0000","last_undegraded":"2023-11-24T08:29:43.918857+0000","last_fullsized":"2023-11-24T08:29:43.918857+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541569+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T08:29:42.541569+0000","last_peered":"2023-11-24T08:29:42.541569+0000","last_clean":"2023-11-24T08:29:42.541569+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T08:29:42.541569+0000","last_undegraded":"2023-11-24T08:29:42.541569+0000","last_fullsized":"2023-11-24T08:29:42.541569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-24T08:29:41.531518+0000","last_change":"2023-11-24T08:29:37.759470+0000","last_active":"2023-11-24T08:29:41.531518+0000","last_peered":"2023-11-24T08:29:41.531518+0000","last_clean":"2023-11-24T08:29:41.531518+0000","last_became_active":"2023-11-24T08:29:37.756694+0000","last_became_peered":"2023-11-24T08:29:37.756694+0000","last_unstale":"2023-11-24T08:29:41.531518+0000","last_undegraded":"2023-11-24T08:29:41.531518+0000","last_fullsized":"2023-11-24T08:29:41.531518+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:28:39.135672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":33732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":716800,"data_stored":624100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:29:46.028 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T08:29:46.297 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:46.298 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T08:29:46.313 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-11-24T08:29:45.946910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82852,"kb_used_data":1528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032668,"statfs":{"total":289910292480,"available":289825452032,"internally_reserved":0,"allocated":1564672,"data_stored":1285410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001558"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541670+0000","last_change":"2023-11-24T08:29:41.544251+0000","last_active":"2023-11-24T08:29:42.541670+0000","last_peered":"2023-11-24T08:29:42.541670+0000","last_clean":"2023-11-24T08:29:42.541670+0000","last_became_active":"2023-11-24T08:29:41.543720+0000","last_became_peered":"2023-11-24T08:29:41.543720+0000","last_unstale":"2023-11-24T08:29:42.541670+0000","last_undegraded":"2023-11-24T08:29:42.541670+0000","last_fullsized":"2023-11-24T08:29:42.541670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:40:21.348430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541291+0000","last_change":"2023-11-24T08:29:41.544774+0000","last_active":"2023-11-24T08:29:42.541291+0000","last_peered":"2023-11-24T08:29:42.541291+0000","last_clean":"2023-11-24T08:29:42.541291+0000","last_became_active":"2023-11-24T08:29:41.543928+0000","last_became_peered":"2023-11-24T08:29:41.543928+0000","last_unstale":"2023-11-24T08:29:42.541291+0000","last_undegraded":"2023-11-24T08:29:42.541291+0000","last_fullsized":"2023-11-24T08:29:42.541291+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:10:33.568413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541469+0000","last_change":"2023-11-24T08:29:41.544569+0000","last_active":"2023-11-24T08:29:42.541469+0000","last_peered":"2023-11-24T08:29:42.541469+0000","last_clean":"2023-11-24T08:29:42.541469+0000","last_became_active":"2023-11-24T08:29:41.543996+0000","last_became_peered":"2023-11-24T08:29:41.543996+0000","last_unstale":"2023-11-24T08:29:42.541469+0000","last_undegraded":"2023-11-24T08:29:42.541469+0000","last_fullsized":"2023-11-24T08:29:42.541469+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:32:34.625958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541423+0000","last_change":"2023-11-24T08:29:41.546688+0000","last_active":"2023-11-24T08:29:42.541423+0000","last_peered":"2023-11-24T08:29:42.541423+0000","last_clean":"2023-11-24T08:29:42.541423+0000","last_became_active":"2023-11-24T08:29:41.546024+0000","last_became_peered":"2023-11-24T08:29:41.546024+0000","last_unstale":"2023-11-24T08:29:42.541423+0000","last_undegraded":"2023-11-24T08:29:42.541423+0000","last_fullsized":"2023-11-24T08:29:42.541423+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:49:39.572701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:43.918845+0000","last_change":"2023-11-24T08:29:41.548070+0000","last_active":"2023-11-24T08:29:43.918845+0000","last_peered":"2023-11-24T08:29:43.918845+0000","last_clean":"2023-11-24T08:29:43.918845+0000","last_became_active":"2023-11-24T08:29:41.547695+0000","last_became_peered":"2023-11-24T08:29:41.547695+0000","last_unstale":"2023-11-24T08:29:43.918845+0000","last_undegraded":"2023-11-24T08:29:43.918845+0000","last_fullsized":"2023-11-24T08:29:43.918845+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:30:28.663728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:43.918960+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T08:29:43.918960+0000","last_peered":"2023-11-24T08:29:43.918960+0000","last_clean":"2023-11-24T08:29:43.918960+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T08:29:43.918960+0000","last_undegraded":"2023-11-24T08:29:43.918960+0000","last_fullsized":"2023-11-24T08:29:43.918960+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:43.918857+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T08:29:43.918857+0000","last_peered":"2023-11-24T08:29:43.918857+0000","last_clean":"2023-11-24T08:29:43.918857+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T08:29:43.918857+0000","last_undegraded":"2023-11-24T08:29:43.918857+0000","last_fullsized":"2023-11-24T08:29:43.918857+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-11-24T08:29:42.541569+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T08:29:42.541569+0000","last_peered":"2023-11-24T08:29:42.541569+0000","last_clean":"2023-11-24T08:29:42.541569+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T08:29:42.541569+0000","last_undegraded":"2023-11-24T08:29:42.541569+0000","last_fullsized":"2023-11-24T08:29:42.541569+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-11-24T08:29:41.531518+0000","last_change":"2023-11-24T08:29:37.759470+0000","last_active":"2023-11-24T08:29:41.531518+0000","last_peered":"2023-11-24T08:29:41.531518+0000","last_clean":"2023-11-24T08:29:41.531518+0000","last_became_active":"2023-11-24T08:29:37.756694+0000","last_became_peered":"2023-11-24T08:29:37.756694+0000","last_unstale":"2023-11-24T08:29:41.531518+0000","last_undegraded":"2023-11-24T08:29:41.531518+0000","last_fullsized":"2023-11-24T08:29:41.531518+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:28:39.135672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":33732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27732,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":716800,"data_stored":624100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T08:29:46.314 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-11-24T08:29:46.314 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-11-24T08:29:46.315 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-11-24T08:29:46.315 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-11-24T08:29:46.610 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T08:29:46.611 INFO:teuthology.orchestra.run.smithi164.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-24T08:29:46.627 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-11-24T08:29:46.628 INFO:teuthology.run_tasks:Running task exec... 2023-11-24T08:29:46.637 INFO:teuthology.task.exec:Executing custom commands... 2023-11-24T08:29:46.637 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T08:29:46.637 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:46.982 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror] 2023-11-24T08:29:46.982 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCxXmBlyVt0FxAAGwZzgfhRCUgDTyUSeeF0jA== 2023-11-24T08:29:46.982 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror-peer" 2023-11-24T08:29:46.982 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror 2023-11-24T08:29:46.999 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:47.365 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2023-11-24T08:29:47.365 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCxXmBl2gf5HBAA9EBJINUJn4ekhd0nGmoufQ== 2023-11-24T08:29:47.365 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2023-11-24T08:29:47.366 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:47.366 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.0 2023-11-24T08:29:47.384 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:47.756 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2023-11-24T08:29:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCxXmBl4z8QIhAA3a9reM8oKsxKBmN1L1M+/w== 2023-11-24T08:29:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2023-11-24T08:29:47.756 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:47.757 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.1 2023-11-24T08:29:47.773 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:48.115 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2023-11-24T08:29:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCxXmBlabBFJxAAPdxB+HC9M8hcmR8+ysxGTQ== 2023-11-24T08:29:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2023-11-24T08:29:48.115 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:48.115 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.2 2023-11-24T08:29:48.133 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:48.473 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2023-11-24T08:29:48.474 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQCxXmBlsqxCLBAA5qBnGHurZD+hEiXrAUafuw== 2023-11-24T08:29:48.474 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2023-11-24T08:29:48.474 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:48.474 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.3 2023-11-24T08:29:48.491 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:48.824 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror] 2023-11-24T08:29:48.824 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBliYkpCRAAXzQ6jLSavAPjhXjZRYC66A== 2023-11-24T08:29:48.824 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror-peer" 2023-11-24T08:29:48.824 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:48.824 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror 2023-11-24T08:29:48.841 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:49.177 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.0] 2023-11-24T08:29:49.178 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlModdDhAA4+Qogd32KBPOVJtOT7ZTHg== 2023-11-24T08:29:49.178 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2023-11-24T08:29:49.178 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:49.178 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.0 2023-11-24T08:29:49.196 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:49.543 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.1] 2023-11-24T08:29:49.543 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlqqybExAAVMrLMjuC+Y/9fwO+IWkLTQ== 2023-11-24T08:29:49.544 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2023-11-24T08:29:49.544 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:49.544 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.1 2023-11-24T08:29:49.562 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:49.902 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.2] 2023-11-24T08:29:49.902 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlFUTLGBAAp8z3EwxyCaLDRcLW2dXkYg== 2023-11-24T08:29:49.902 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2023-11-24T08:29:49.902 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.2 2023-11-24T08:29:49.919 DEBUG:teuthology.orchestra.run.smithi164:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-24T08:29:50.271 INFO:teuthology.orchestra.run.smithi164.stdout:[client.mirror.3] 2023-11-24T08:29:50.272 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQDWXmBlq1ehHRAAL56tB63zg/r/YSWQDCnMoA== 2023-11-24T08:29:50.272 INFO:teuthology.orchestra.run.smithi164.stdout: caps mon = "profile rbd-mirror" 2023-11-24T08:29:50.272 INFO:teuthology.orchestra.run.smithi164.stdout: caps osd = "profile rbd" 2023-11-24T08:29:50.272 INFO:teuthology.orchestra.run.smithi164.stderr:updated caps for client.mirror.3 2023-11-24T08:29:50.292 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-24T08:29:50.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-24T08:29:50.302 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-24T08:29:50.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-24T08:29:50.304 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-24T08:29:50.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-24T08:29:50.323 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-24T08:29:50.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-24T08:29:50.325 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-24T08:29:50.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-11-24T08:29:50.334 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-11-24T08:29:50.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-11-24T08:29:50.336 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-24T08:29:50.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-11-24T08:29:50.345 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-11-24T08:29:50.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-11-24T08:29:50.347 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-24T08:29:50.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-11-24T08:29:50.356 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-11-24T08:29:50.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-11-24T08:29:50.358 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-24T08:29:50.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-24T08:29:50.367 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-24T08:29:50.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-24T08:29:50.370 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-24T08:29:50.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-11-24T08:29:50.380 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-11-24T08:29:50.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-11-24T08:29:50.382 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-24T08:29:50.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-24T08:29:50.391 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-24T08:29:50.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-24T08:29:50.394 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-11-24T08:29:50.409 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1700814590 2023-11-24T08:29:50.410 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-11-24T08:29:50.410 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-11-24T08:29:50.418 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1700814590 2023-11-24T08:29:50.419 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-11-24T08:29:50.419 INFO:teuthology.run_tasks:Running task workunit... 2023-11-24T08:29:50.433 INFO:tasks.workunit:Pulling workunits from ref 135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 2023-11-24T08:29:50.433 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-24T08:29:50.434 DEBUG:teuthology.orchestra.run.smithi164:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-11-24T08:29:50.434 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-11-24T08:29:50.434 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2023-11-24T08:29:50.434 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-11-24T08:29:50.435 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-11-24T08:29:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T08:29:50.439 INFO:teuthology.orchestra.run.smithi164.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-11-24T08:29:50.440 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-11-24T08:29:50.484 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-11-24T08:29:50.484 DEBUG:teuthology.orchestra.run.smithi164:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-11-24T08:29:50.533 INFO:tasks.workunit:timeout=6h 2023-11-24T08:29:50.533 INFO:tasks.workunit:cleanup=True 2023-11-24T08:29:50.533 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 2023-11-24T08:29:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-11-24T08:31:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Note: switching to '135a1ac939fb4e6d1ef988cf89d0da570e04d3d1'. 2023-11-24T08:31:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:state without impacting any branches by switching back to a branch. 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: git switch -c 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Or undo this operation with: 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:31:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: git switch - 2023-11-24T08:31:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:31:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-24T08:31:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:31:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:HEAD is now at 135a1ac939f Merge branch 'wip-63607' of https://github.com/idryomov/ceph into HEAD 2023-11-24T08:31:12.400 DEBUG:teuthology.orchestra.run.smithi164:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-11-24T08:31:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-24T08:31:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-11-24T08:31:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-24T08:31:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-24T08:31:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-24T08:31:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-11-24T08:31:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-11-24T08:31:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-24T08:31:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-11-24T08:31:13.402 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2023-11-24T08:31:13.402 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-11-24T08:31:13.448 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-11-24T08:31:13.449 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-11-24T08:31:13.449 DEBUG:teuthology.orchestra.run.smithi164:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=135a1ac939fb4e6d1ef988cf89d0da570e04d3d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-11-24T08:31:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ IMAGE_COUNT=50 2023-11-24T08:31:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ export LOCKDEP=0 2023-11-24T08:31:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-11-24T08:31:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ type xmlstarlet 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ XMLSTARLET=xmlstarlet 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ RBD_MIRROR_INSTANCES=4 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CLUSTER1=cluster1 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CLUSTER2=cluster2 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ PEER_CLUSTER_SUFFIX= 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ POOL=mirror 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ PARENT_POOL=mirror_parent 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ NS1=ns1 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ NS2=ns2 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ TEMPDIR= 2023-11-24T08:31:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ID=mirror 2023-11-24T08:31:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2023-11-24T08:31:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ MIRROR_USER_ID_PREFIX=mirror. 2023-11-24T08:31:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ RBD_MIRROR_MODE=snapshot 2023-11-24T08:31:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ MIRROR_POOL_MODE=image 2023-11-24T08:31:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:31:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ MIRROR_POOL_MODE=image 2023-11-24T08:31:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ export CEPH_ARGS=--id mirror 2023-11-24T08:31:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ LAST_MIRROR_INSTANCE=3 2023-11-24T08:31:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-11-24T08:31:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-11-24T08:31:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-11-24T08:31:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_BIN=. 2023-11-24T08:31:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_SRC=. 2023-11-24T08:31:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -e CMakeCache.txt ] 2023-11-24T08:31:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -gt 0 ] 2023-11-24T08:31:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ setup 2023-11-24T08:31:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local c 2023-11-24T08:31:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ trap cleanup $? INT TERM EXIT 2023-11-24T08:31:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ setup_tempdir 2023-11-24T08:31:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mktemp -d 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ TEMPDIR=/tmp/tmp.tyy3Nh9UuT 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -z 1 ] 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ setup_pools cluster1 cluster2 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster2 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mon_map_file 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mon_addr 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local admin_key_file 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid 2023-11-24T08:31:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror 64 64 2023-11-24T08:31:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror' already exists 2023-11-24T08:31:14.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-11-24T08:31:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror_parent' already exists 2023-11-24T08:31:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror 2023-11-24T08:31:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror_parent 2023-11-24T08:31:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-11-24T08:31:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-11-24T08:31:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-11-24T08:31:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-11-24T08:31:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-11-24T08:31:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-11-24T08:31:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -z ] 2023-11-24T08:31:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -z ] 2023-11-24T08:31:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_add cluster1 mirror cluster2 2023-11-24T08:31:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local client_cluster=cluster2 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster2 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid_var_name 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local peer_uuid 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:31:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_uuid=2d4ea7bb-22ab-4416-b1e1-a24a55f577b7 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 17 ] 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 -eq 0 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror_parent 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local client_cluster=cluster2 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster2 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid_var_name 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local peer_uuid 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:31:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-11-24T08:31:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_uuid=dcba6e1a-da07-4157-86f8-90ef58d6c5b8 2023-11-24T08:31:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T08:31:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:31:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 17 ] 2023-11-24T08:31:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 -eq 0 2023-11-24T08:31:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:31:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ setup_pools cluster2 cluster1 2023-11-24T08:31:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:31:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster1 2023-11-24T08:31:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mon_map_file 2023-11-24T08:31:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mon_addr 2023-11-24T08:31:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local admin_key_file 2023-11-24T08:31:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid 2023-11-24T08:31:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror 64 64 2023-11-24T08:31:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror' already exists 2023-11-24T08:31:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-11-24T08:31:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror_parent' already exists 2023-11-24T08:31:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror 2023-11-24T08:31:26.436 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7013718028604047) > weight (0.25) 2023-11-24T08:31:26.436 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6887031952728958) > weight (0.25) 2023-11-24T08:31:26.436 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4684630801988574) > weight (0.25) 2023-11-24T08:31:26.436 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6805157652145022) > weight (0.25) 2023-11-24T08:31:26.436 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2023-11-24T08:31:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror_parent 2023-11-24T08:31:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-11-24T08:31:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-11-24T08:31:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-11-24T08:31:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-11-24T08:31:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-11-24T08:31:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-11-24T08:31:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -z ] 2023-11-24T08:31:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -z ] 2023-11-24T08:31:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_add cluster2 mirror cluster1 2023-11-24T08:31:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:31:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local client_cluster=cluster1 2023-11-24T08:31:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster1 2023-11-24T08:31:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid_var_name 2023-11-24T08:31:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code 2023-11-24T08:31:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local peer_uuid 2023-11-24T08:31:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:31:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-11-24T08:31:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_uuid=3d28e6c8-cd7a-4af0-a950-af605eb09c93 2023-11-24T08:31:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T08:31:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:31:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 17 ] 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 -eq 0 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror_parent 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local client_cluster=cluster1 2023-11-24T08:31:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local remote_cluster=cluster1 2023-11-24T08:31:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local uuid_var_name 2023-11-24T08:31:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code 2023-11-24T08:31:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local peer_uuid 2023-11-24T08:31:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:31:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-11-24T08:31:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ peer_uuid=dc1c9e8c-3321-4ffa-8219-1fcd6faafc9e 2023-11-24T08:31:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T08:31:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:31:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 17 ] 2023-11-24T08:31:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 -eq 0 2023-11-24T08:31:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:31:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:31:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirrors cluster1 2023-11-24T08:31:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:31:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1:0 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1:1 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1:2 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T08:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1:3 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirrors cluster2 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster2:0 2023-11-24T08:31:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:0 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster2:0 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster2:0 ] 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster2 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster2:1 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:1 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster2:1 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster2:1 ] 2023-11-24T08:31:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster2 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster2:2 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:2 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster2:2 2023-11-24T08:31:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster2:2 ] 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster2 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T08:31:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster2:3 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:3 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster2:3 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster2:3 ] 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster2 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster2 2023-11-24T08:31:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog TEST: add image and test replay after client crashes 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.tyy3Nh9UuT/rbd-mirror.test.log 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ date +%F %T 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2023-11-24 08:31:29 TEST: add image and test replay after client crashes 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2023-11-24 08:31:29 TEST: add image and test replay after client crashes 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=test 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:31:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror test 512M 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:31:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T08:31:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T08:31:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2023-11-24T08:31:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:31:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-11-24T08:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:31:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-11-24T08:31:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:31:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-11-24T08:31:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:31:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10abe0128b3b 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10abe0128b3b 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:29 2023 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:31:29 2023 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:29 2023 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:31:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:31:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:31:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:31:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:31:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:31:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster=cluster1 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:31:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep+ local -i loop_instance state 2023-11-24T08:31:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:31:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:31:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:31:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:31:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:31:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:31:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:31.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:31:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:31:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:31:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:31:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:31:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:31:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:31:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:31:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:31:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:31:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:31:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:31:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:31:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:31:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:31:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:31:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:31:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:31:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:31:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:31:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:31:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:31:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:31:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:31:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:31:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:31:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:31:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:31:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:31:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:31:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:31:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:31:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:31:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:31:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:31:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:31:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:31:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:31:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:31:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:31:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:31:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:31:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:31:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:31:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:31:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:31:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:31:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:31:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:31:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:31:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:31:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:31:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:31:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:31:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:31:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:31:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:31:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:31:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:31:35.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:31:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:31:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:31:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:31:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:31:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:31:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:31:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:31:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:31:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:31:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:31:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:31:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:31:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T08:31:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:31:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:31:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:31:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:31:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:31:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:31:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:31:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:31:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:31:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:31:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:31:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:31:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:31:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:31:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:31:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:31:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:31:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:31:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:31:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:31:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:31:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:31:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:31:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:31:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:31:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:31:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:31:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:31:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:31:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:31:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:31:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:31:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:31:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:31:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:31:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:31:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:31:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:31:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:31:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:31:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:31:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:31:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:31:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:31:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:31:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:31:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:31:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:31:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:31:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:31:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:31:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:31:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:31:41.435 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-24T08:31:41.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-24T08:31:41.436 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-24T08:31:41.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T08:31:41.434+0000 7f37ee037640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39807) UID: 1000 2023-11-24T08:31:44.436 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-24T08:31:44.437 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:31:44.437 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-24T08:31:44.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-24T08:31:44.437 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-24T08:31:44.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-24T08:31:44.441 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-11-24T08:31:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:31:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:31:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:31:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:31:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:31:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:31:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:31:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:31:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:31:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:31:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:31:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:31:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:31:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:31:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:31:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:31:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:31:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:31:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:31:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:31:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:31:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:31:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:31:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:31:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:31:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:31:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:31:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:31:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:31:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:31:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:31:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:31:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name= 2023-11-24T08:31:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 10 2023-11-24T08:31:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:31:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:31:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:31:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T08:31:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:31:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:31:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 17996800 bytes 2023-11-24T08:31:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 29857792 bytes 2023-11-24T08:31:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 57792512 bytes 2023-11-24T08:31:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 125725696 bytes 2023-11-24T08:31:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 231758848 bytes 2023-11-24T08:31:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 259849216 bytes 2023-11-24T08:31:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 260463616 bytes 2023-11-24T08:31:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 319463424 bytes 2023-11-24T08:31:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 357710848 bytes 2023-11-24T08:31:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 366050304 bytes 2023-11-24T08:31:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 427635712 bytes 2023-11-24T08:31:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 473109504 bytes 2023-11-24T08:31:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 527433728 bytes 2023-11-24T08:31:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 563449856 bytes 2023-11-24T08:31:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 605569024 bytes 2023-11-24T08:31:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 668289024 bytes 2023-11-24T08:31:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 673856512 bytes 2023-11-24T08:31:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 705246208 bytes 2023-11-24T08:31:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 740808704 bytes 2023-11-24T08:31:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 781293568 bytes 2023-11-24T08:31:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 846855168 bytes 2023-11-24T08:31:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 925024256 bytes 2023-11-24T08:31:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 964296704 bytes 2023-11-24T08:31:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1039284224 bytes 2023-11-24T08:31:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1071832064 bytes 2023-11-24T08:31:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1072447488 bytes 2023-11-24T08:31:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1092121600 bytes 2023-11-24T08:31:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1162944512 bytes 2023-11-24T08:31:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1210999808 bytes 2023-11-24T08:31:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1233409024 bytes 2023-11-24T08:31:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1247150080 bytes 2023-11-24T08:31:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1297720320 bytes 2023-11-24T08:31:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1374238720 bytes 2023-11-24T08:31:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1415845888 bytes 2023-11-24T08:31:54.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1444800512 bytes 2023-11-24T08:31:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1466120192 bytes 2023-11-24T08:31:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1531062272 bytes 2023-11-24T08:31:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1559002112 bytes 2023-11-24T08:31:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1615683584 bytes 2023-11-24T08:31:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1672168448 bytes 2023-11-24T08:31:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1690441728 bytes 2023-11-24T08:31:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1711908864 bytes 2023-11-24T08:31:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1799041024 bytes 2023-11-24T08:31:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:31:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:31:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:31:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:31:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:31:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:31:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:31:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:31:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:31:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:31:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:31:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:31:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:31:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:31:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:31:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:31:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:31:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:31:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:31:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:31:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:31:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2023-11-24T08:31:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2023-11-24T08:31:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24T08:31:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:31:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:31:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:31:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:31:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:31:52 2023-11-24T08:31:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:31:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 1 2023-11-24T08:31:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:31:59.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:31:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:31:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:31:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:31:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:31:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:31:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2023-11-24T08:31:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2023-11-24T08:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24T08:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:31:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:31:52 2023-11-24T08:31:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:31:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 1 2023-11-24T08:31:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:32:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:32:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:32:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:32:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:32:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2023-11-24T08:32:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2023-11-24T08:32:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24T08:32:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:32:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:32:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:32:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:32:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:31:52 2023-11-24T08:32:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 1 2023-11-24T08:32:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:32:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:32:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:32:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:32:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:32:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:32:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:32:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:32:11.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:32:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:32:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:32:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2023-11-24T08:32:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2023-11-24T08:32:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24T08:32:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:32:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:32:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:32:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:32:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:31:52 2023-11-24T08:32:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 1 2023-11-24T08:32:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:32:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:32:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:32:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:32:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:32:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:32:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:32:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1700814690,"remote_snapshot_timestamp":1700814690,"replay_state":"idle"} 2023-11-24T08:32:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.0 on smithi164 2023-11-24T08:32:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:32:14 2023-11-24T08:32:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:32:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:32:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:32:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:32:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:31:52 2023-11-24T08:32:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:32:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1700814690,"remote_snapshot_timestamp":1700814690,"replay_state":"idle"} 2023-11-24T08:32:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:32:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:32:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^up+ 2023-11-24T08:32:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:32:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.0 on smithi164 2023-11-24T08:32:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:32:14 2023-11-24T08:32:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:32:14 2023-11-24T08:32:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:32:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:32:14 2023-11-24T08:32:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:32:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror.mirror_status 2023-11-24T08:32:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:32:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:32:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:32:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:32:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:32:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:32:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:32:14 2023-11-24T08:32:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:32:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:32:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:32:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:32:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:32:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1 2023-11-24T08:32:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap1 2023-11-24T08:32:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:32:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1 2023-11-24T08:32:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:32:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:32:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:32:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:32:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:32:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:32:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:32:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:32:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:32:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:32:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status+ mirror/test 2023-11-24T08:32:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i state 2023-11-24T08:32:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:local cluster_instance=cluster1 2023-11-24T08:32:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:32:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:32:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:32:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:32:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:32:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:32:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:32:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:32:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:32:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:32:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:32:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:32:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:32:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:32:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:32:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:32:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:32:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:32:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:32:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:32:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:32:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:32:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:32:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:32:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:32:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:32:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:32:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:32:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:32:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:32:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:32:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:32:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:32:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:32:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:32:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:32:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:32:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:32:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:32:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:32:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:32:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + grep -i Replaying 2023-11-24T08:32:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo "state": "Replaying" 2023-11-24T08:32:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:32:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:32:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:32:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:32:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:32:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:32:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:32:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:32:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:32:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:32:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:32:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:32:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:32:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:32:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:32:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:32:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:32:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:32:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:32:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:32:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:32:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 6 2023-11-24T08:32:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:32:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:32:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:32:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:32:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:32:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:32:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=6 2023-11-24T08:32:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:32:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:32:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:32:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:32:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:32:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=4 2023-11-24T08:32:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 6 = 4 2023-11-24T08:32:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:32:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:32:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:32:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:32:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=4 2023-11-24T08:32:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 6 = 4 2023-11-24T08:32:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:32:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:32:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:32:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:32:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:32:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:32:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:32:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:32:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:32:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 6 = 2023-11-24T08:32:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:32:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:32:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:32:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:32:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:32:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:32:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:32:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:32:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:32:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 6 = 2023-11-24T08:32:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:32:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:32:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:32:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:32:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=6 2023-11-24T08:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 6 = 6 2023-11-24T08:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-11-24T08:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1 2023-11-24T08:32:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:32:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:32:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-11-24T08:32:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:32:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:32:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:32:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:32:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:32:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:32:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:32:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:32:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:32:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:32:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap1 2023-11-24T08:32:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1 2023-11-24T08:32:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:32:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:32:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:32:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:32:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:32:34.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:32:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:32:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:32:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:32:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:32:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap1-clean 2023-11-24T08:32:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-11-24T08:32:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:32:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1-clean 2023-11-24T08:32:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:32:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:32:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:32:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:32:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2023-11-24T08:32:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:32:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:32:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 58057728 bytes 2023-11-24T08:32:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 96889856 bytes 2023-11-24T08:32:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 143587328 bytes 2023-11-24T08:32:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 223876096 bytes 2023-11-24T08:32:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 300079104 bytes 2023-11-24T08:32:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 309695488 bytes 2023-11-24T08:32:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 337452032 bytes 2023-11-24T08:32:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 391412736 bytes 2023-11-24T08:32:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 404792320 bytes 2023-11-24T08:32:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 451174400 bytes 2023-11-24T08:32:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 493933568 bytes 2023-11-24T08:32:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 569238528 bytes 2023-11-24T08:32:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 642801664 bytes 2023-11-24T08:32:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 708599808 bytes 2023-11-24T08:32:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 742599680 bytes 2023-11-24T08:32:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 787773440 bytes 2023-11-24T08:32:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 801517568 bytes 2023-11-24T08:32:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 921076736 bytes 2023-11-24T08:32:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 927004672 bytes 2023-11-24T08:32:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 977790976 bytes 2023-11-24T08:32:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1017195520 bytes 2023-11-24T08:32:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1089372160 bytes 2023-11-24T08:32:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1181798400 bytes 2023-11-24T08:32:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1231243264 bytes 2023-11-24T08:32:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1276275712 bytes 2023-11-24T08:32:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1347516416 bytes 2023-11-24T08:32:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1423277056 bytes 2023-11-24T08:32:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1504690176 bytes 2023-11-24T08:32:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1571270656 bytes 2023-11-24T08:32:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1611034624 bytes 2023-11-24T08:32:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1642613760 bytes 2023-11-24T08:32:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1732155392 bytes 2023-11-24T08:32:43.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1792865280 bytes 2023-11-24T08:32:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1852773376 bytes 2023-11-24T08:32:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1923714048 bytes 2023-11-24T08:32:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1961111552 bytes 2023-11-24T08:32:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1994068992 bytes 2023-11-24T08:32:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 2063254528 bytes 2023-11-24T08:32:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 2094514176 bytes 2023-11-24T08:32:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 2117893120 bytes 2023-11-24T08:32:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 2143384576 bytes 2023-11-24T08:32:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2148139008 bytes 2023-11-24T08:32:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2228807680 bytes 2023-11-24T08:32:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2242515968 bytes 2023-11-24T08:32:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2274500608 bytes 2023-11-24T08:32:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2295865344 bytes 2023-11-24T08:32:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2390851584 bytes 2023-11-24T08:32:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2451357696 bytes 2023-11-24T08:32:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2499941376 bytes 2023-11-24T08:32:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2525102080 bytes 2023-11-24T08:32:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:32:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:32:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:32:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:32:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:32:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:32:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:32:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:32:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:32:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:32:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:32:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:32:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:32:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:32:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:32:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:32:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:47.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":17210504.53,"bytes_per_snapshot":258157568.0,"last_snapshot_bytes":516315136,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814747,"remote_snapshot_timestamp":1700814747,"replay_state":"idle"} 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.0 on smithi164 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:32:44 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:32:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:32:20 2023-11-24T08:32:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:32:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":17210504.53,"bytes_per_snapshot":258157568.0,"last_snapshot_bytes":516315136,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814747,"remote_snapshot_timestamp":1700814747,"replay_state":"idle"} 2023-11-24T08:32:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:32:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:32:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^up+ 2023-11-24T08:32:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:32:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.0 on smithi164 2023-11-24T08:32:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:32:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:32:44 2023-11-24T08:32:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:32:44 2023-11-24T08:32:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:32:44 2023-11-24T08:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror.mirror_status 2023-11-24T08:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:32:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:32:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:32:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:32:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:32:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:32:44 2023-11-24T08:32:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:32:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:32:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:32:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:32:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2 2023-11-24T08:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap2 2023-11-24T08:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2 2023-11-24T08:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:32:50.442 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-24T08:32:50.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-24T08:32:50.442 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-24T08:32:50.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T08:32:50.441+0000 7f59f5268640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 40677) UID: 1000 2023-11-24T08:32:51.443 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-24T08:32:51.444 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:32:51.444 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-24T08:32:51.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-24T08:32:51.444 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-24T08:32:51.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-24T08:32:51.447 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2023-11-24T08:33:13.436 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.26121130067866627) > weight (0.25) 2023-11-24T08:33:13.437 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-24T08:33:13.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-24T08:33:13.437 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-24T08:33:13.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T08:33:13.433+0000 7f31a2b38640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 39819) UID: 1000 2023-11-24T08:33:15.438 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-24T08:33:15.439 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:33:15.439 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-24T08:33:15.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-24T08:33:15.440 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-24T08:33:15.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-24T08:33:15.443 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-11-24T08:33:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:33:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:33:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:33:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:33:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:33:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:33:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:33:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:33:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:33:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:33:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:33:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:33:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:33:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sleep 0 2023-11-24T08:33:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:33:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:33:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:33:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:33:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:33:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:33:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:33:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:33:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:33:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:33:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:33:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:33:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:33:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:33:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:33:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:33:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:33:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:33:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:33:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:16.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:33:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:33:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:33:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:33:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:33:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:33:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:33:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:33:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:33:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:33:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:33:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:33:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:33:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:33:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:33:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:33:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:33:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:33:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:18.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:33:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:33:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:33:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:33:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:33:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:33:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:33:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:33:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:33:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:33:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:33:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:33:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:33:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:33:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:33:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:33:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:33:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:33:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:33:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:33:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:33:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:33:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:33:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:33:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:33:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:33:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:33:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:33:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:33:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:33:20.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:33:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:33:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:33:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:33:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:33:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:33:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:33:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:33:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:33:21.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:21.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:33:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:33:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:33:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:33:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:33:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:33:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:21.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:33:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:33:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:33:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:33:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:33:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:33:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:33:21.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:33:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:33:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:33:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:33:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:33:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T08:33:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:33:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:33:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:33:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:33:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:33:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:33:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:33:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:33:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:33:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:25.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:33:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:33:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:33:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:33:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:33:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:33:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:33:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:33:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:33:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:33:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:33:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:33:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:33:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:33:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:33:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:33:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:33:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:33:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:33:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:33:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:33:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:33:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:33:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:33:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:33:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:33:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:33:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:33:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:33:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:33:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:33:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:33:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:33:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:33:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:33:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:33:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:33:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + grep -i Replaying 2023-11-24T08:33:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo "state": "Replaying" 2023-11-24T08:33:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:33:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:33:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:33:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:33:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:33:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:33:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:33:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 9 2023-11-24T08:33:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:33:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:33:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:33:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:33:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:33:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:33:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=9 2023-11-24T08:33:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:33:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:33:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:33:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:33:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:33:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:33:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:33:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:33:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:33:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:33:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:33:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:33:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 9 = 2023-11-24T08:33:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:33:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:33:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:33:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:33:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:33:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:33:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:33:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:33:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:33:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:33:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:33:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 9 = 2023-11-24T08:33:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:33:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:33:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:33:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:33:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:33:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:33:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:33:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:33:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:33:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:33:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:33:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 9 = 2023-11-24T08:33:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:33:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:33:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:33:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:33:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:33:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=9 2023-11-24T08:33:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 9 = 9 2023-11-24T08:33:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:33:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-11-24T08:33:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2 2023-11-24T08:33:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:33:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:33:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-11-24T08:33:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:33:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:33:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:33:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 4 2023-11-24T08:33:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:33:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:33:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:33:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:33:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:33:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snap1-clean ] 2023-11-24T08:33:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap1-clean 2023-11-24T08:33:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1-clean 2023-11-24T08:33:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:33:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:33:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:33:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:33:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:33:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap2 2023-11-24T08:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2 2023-11-24T08:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:33:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:33:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:33:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:33:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:33:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap2-clean 2023-11-24T08:33:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-11-24T08:33:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:33:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2-clean 2023-11-24T08:33:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:33:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:33:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:33:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:33:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2023-11-24T08:33:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:33:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:33:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 89452544 bytes 2023-11-24T08:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 114625536 bytes 2023-11-24T08:33:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 162449408 bytes 2023-11-24T08:33:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 209098752 bytes 2023-11-24T08:33:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 228882432 bytes 2023-11-24T08:33:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 229496832 bytes 2023-11-24T08:33:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 292148224 bytes 2023-11-24T08:33:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 337011712 bytes 2023-11-24T08:33:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 362948608 bytes 2023-11-24T08:33:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 375298048 bytes 2023-11-24T08:33:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 397206528 bytes 2023-11-24T08:33:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 414206976 bytes 2023-11-24T08:33:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 453090304 bytes 2023-11-24T08:33:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 483047424 bytes 2023-11-24T08:33:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 511921152 bytes 2023-11-24T08:33:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 512543744 bytes 2023-11-24T08:33:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 604477440 bytes 2023-11-24T08:33:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 624886784 bytes 2023-11-24T08:33:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 665205760 bytes 2023-11-24T08:33:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 764195840 bytes 2023-11-24T08:33:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 789468160 bytes 2023-11-24T08:33:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 797711360 bytes 2023-11-24T08:33:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 840645632 bytes 2023-11-24T08:33:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 857333760 bytes 2023-11-24T08:33:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 930627584 bytes 2023-11-24T08:33:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 976022528 bytes 2023-11-24T08:33:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1032383488 bytes 2023-11-24T08:33:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1113108480 bytes 2023-11-24T08:33:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1129858048 bytes 2023-11-24T08:33:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1189560320 bytes 2023-11-24T08:33:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1224116224 bytes 2023-11-24T08:33:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1258766336 bytes 2023-11-24T08:33:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1358305280 bytes 2023-11-24T08:33:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1408333824 bytes 2023-11-24T08:33:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1413905408 bytes 2023-11-24T08:33:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1432603648 bytes 2023-11-24T08:33:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1478450176 bytes 2023-11-24T08:33:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:33:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:33:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:33:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:33:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:33:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:33:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:33:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:33:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:33:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:33:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:33:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:33:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:33:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:33:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:33:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:33:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1700814747,"remote_snapshot_timestamp":1700814747,"replay_state":"idle"} 2023-11-24T08:33:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2023-11-24T08:33:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:33:23 2023-11-24T08:33:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:33:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:33:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:33:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:33:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:33:20 2023-11-24T08:33:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:33:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:33:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:33:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1700814747,"remote_snapshot_timestamp":1700814747,"replay_state":"idle"} 2023-11-24T08:33:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:33:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:33:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^up+ 2023-11-24T08:33:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:33:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:33:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2023-11-24T08:33:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:33:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:33:23 2023-11-24T08:33:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:33:23 2023-11-24T08:33:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:33:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:33:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:33:23 2023-11-24T08:33:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:33:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sed -e s|[/ ]|_|g 2023-11-24T08:33:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo cluster1-mirror.mirror_status 2023-11-24T08:33:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:33:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:33:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:33:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:33:44.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:33:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:33:23 2023-11-24T08:33:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:33:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:33:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:33:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:33:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:33:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:33:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:33:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3 2023-11-24T08:33:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap3 2023-11-24T08:33:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:33:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:33:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:33:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3 2023-11-24T08:33:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:34:10.448 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5223211344803304) > weight (0.25) 2023-11-24T08:34:10.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9402561721358892) > weight (0.25) 2023-11-24T08:34:10.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6014783778934925) > weight (0.25) 2023-11-24T08:34:10.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.939921756163639) > weight (0.25) 2023-11-24T08:34:10.449 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2023-11-24T08:34:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:34:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:34:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:34:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:34:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:34:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:34:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:34:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:34:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:34:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:34:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:34:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:34:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:34:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:34:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:34:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:34:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:34:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:34:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:34:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:34:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:34:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:34:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:34:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:34:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:34:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:34:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:34:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:34:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:34:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:34:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:34:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:34:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:34:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:34:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:34:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:34:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:34:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:34:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:34:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:34:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:34:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:34:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:34:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:34:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:34:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:34:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:34:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:34:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:34:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:34:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:34:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:34:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:34:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:34:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:34:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:34:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:34:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:34:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:34:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:34:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:34:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:34:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:34:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:34:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:34:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:34:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:34:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:34:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:34:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:34:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:34:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:34:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:34:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:34:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:34:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:34:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:34:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:34:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:16.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:34:16.443 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.44023138950940266) > weight (0.25) 2023-11-24T08:34:16.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8496033245915566) > weight (0.25) 2023-11-24T08:34:16.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5191485584102564) > weight (0.25) 2023-11-24T08:34:16.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6622895820568899) > weight (0.25) 2023-11-24T08:34:16.444 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2023-11-24T08:34:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 13 2023-11-24T08:34:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:34:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:34:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:34:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:34:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:34:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:34:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=13 2023-11-24T08:34:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:34:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:34:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:34:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:34:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:34:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:34:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:34:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:34:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:34:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:34:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 13 = 2023-11-24T08:34:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:34:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:34:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:34:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:34:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:34:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:34:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:34:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:34:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:34:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:34:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:34:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 13 = 2023-11-24T08:34:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:34:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:34:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:34:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:34:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:34:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:34:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:34:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:34:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:34:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:34:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:34:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 13 = 2023-11-24T08:34:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:34:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:34:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:34:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:34:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:34:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:34:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=13 2023-11-24T08:34:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 13 = 13 2023-11-24T08:34:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:34:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-11-24T08:34:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3 2023-11-24T08:34:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:34:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:34:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-11-24T08:34:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:34:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:34:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:34:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 6 2023-11-24T08:34:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:33:32 2023 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:34:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:34:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:34:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:34:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snap2-clean ] 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap2-clean 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2-clean 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:34:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:34:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:34:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:34:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap3 2023-11-24T08:34:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3 2023-11-24T08:34:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:34:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:34:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:34:25.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:34:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:34:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:34:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:34:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap3-clean 2023-11-24T08:34:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-11-24T08:34:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:34:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3-clean 2023-11-24T08:34:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:34:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:34:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:34:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:34:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T08:34:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:34:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:34:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 41578496 bytes 2023-11-24T08:34:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 101943296 bytes 2023-11-24T08:34:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 161782784 bytes 2023-11-24T08:34:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 200284160 bytes 2023-11-24T08:34:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 226609152 bytes 2023-11-24T08:34:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 231508992 bytes 2023-11-24T08:34:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 268435456 bytes 2023-11-24T08:34:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 289106944 bytes 2023-11-24T08:34:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 335080448 bytes 2023-11-24T08:34:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 356989952 bytes 2023-11-24T08:34:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 402265088 bytes 2023-11-24T08:34:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 429097984 bytes 2023-11-24T08:34:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 481439744 bytes 2023-11-24T08:34:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 518036480 bytes 2023-11-24T08:34:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 568019968 bytes 2023-11-24T08:34:30.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 644511744 bytes 2023-11-24T08:34:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 700092416 bytes 2023-11-24T08:34:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 728509440 bytes 2023-11-24T08:34:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 889590784 bytes 2023-11-24T08:34:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 941499392 bytes 2023-11-24T08:34:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 942112768 bytes 2023-11-24T08:34:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 995055616 bytes 2023-11-24T08:34:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1031719936 bytes 2023-11-24T08:34:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1093748736 bytes 2023-11-24T08:34:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1138717696 bytes 2023-11-24T08:34:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1170023424 bytes 2023-11-24T08:34:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1236250624 bytes 2023-11-24T08:34:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1260264448 bytes 2023-11-24T08:34:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1264258048 bytes 2023-11-24T08:34:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1314266112 bytes 2023-11-24T08:34:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1352411136 bytes 2023-11-24T08:34:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1437588480 bytes 2023-11-24T08:34:33.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1455260672 bytes 2023-11-24T08:34:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1459395584 bytes 2023-11-24T08:34:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1531128832 bytes 2023-11-24T08:34:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1633849344 bytes 2023-11-24T08:34:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1667431424 bytes 2023-11-24T08:34:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1725755392 bytes 2023-11-24T08:34:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1764960256 bytes 2023-11-24T08:34:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1814569984 bytes 2023-11-24T08:34:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1861159936 bytes 2023-11-24T08:34:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1885029376 bytes 2023-11-24T08:34:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1913211904 bytes 2023-11-24T08:34:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2017879040 bytes 2023-11-24T08:34:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2031761408 bytes 2023-11-24T08:34:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2068838400 bytes 2023-11-24T08:34:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2089859072 bytes 2023-11-24T08:34:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2126137344 bytes 2023-11-24T08:34:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2151485440 bytes 2023-11-24T08:34:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2156028928 bytes 2023-11-24T08:34:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2221315072 bytes 2023-11-24T08:34:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2230169600 bytes 2023-11-24T08:34:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2258910208 bytes 2023-11-24T08:34:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2292842496 bytes 2023-11-24T08:34:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2311298048 bytes 2023-11-24T08:34:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2347951104 bytes 2023-11-24T08:34:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2368250880 bytes 2023-11-24T08:34:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2411018240 bytes 2023-11-24T08:34:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2479443968 bytes 2023-11-24T08:34:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2549783552 bytes 2023-11-24T08:34:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:34:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:34:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:34:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:34:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:34:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:34:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:34:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:34:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:34:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:34:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:34:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:34:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:34:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:34:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:34:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:34:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:34:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:34:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:34:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":13815910.4,"bytes_per_snapshot":514832896.0,"last_snapshot_bytes":498624512,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814806,"remote_snapshot_timestamp":1700814856,"replay_state":"syncing","seconds_until_synced":37,"syncing_percent":100,"syncing_snapshot_timestamp":1700814856} 2023-11-24T08:34:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2023-11-24T08:34:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:34:20 2023-11-24T08:34:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:34:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:34:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:34:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:34:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:34:20 2023-11-24T08:34:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:34:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:34:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:34:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":13815910.4,"bytes_per_snapshot":514832896.0,"last_snapshot_bytes":498624512,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814806,"remote_snapshot_timestamp":1700814856,"replay_state":"syncing","seconds_until_synced":37,"syncing_percent":100,"syncing_snapshot_timestamp":1700814856} 2023-11-24T08:34:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:34:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:34:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^up+ 2023-11-24T08:34:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:34:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:34:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2023-11-24T08:34:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:34:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:34:20 2023-11-24T08:34:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:34:20 2023-11-24T08:34:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:34:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:34:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:34:20 2023-11-24T08:34:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:34:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo+ cluster1-mirror.mirror_status 2023-11-24T08:34:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sed -e s|[/ ]|_|g 2023-11-24T08:34:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:34:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:34:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:34:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:34:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:34:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:34:20 2023-11-24T08:34:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:34:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:34:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:34:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:34:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:34:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:34:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:34:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4 2023-11-24T08:34:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap4 2023-11-24T08:34:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:34:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:34:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:34:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4 2023-11-24T08:34:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:35:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:35:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:35:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:35:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:35:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:35:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:35:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:35:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:35:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:35:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:35:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:35:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:35:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:35:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:35:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:35:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i state 2023-11-24T08:35:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:35:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:35:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:35:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:35:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:35:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:35:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:35:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:35:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:35:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:35:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:35:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:35:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:35:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:35:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:35:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:35:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:35:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:35:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:35:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:35:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:35:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:35:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:35:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:35:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:35:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:35:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:35:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:35:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:35:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:35:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:35:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:35:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:35:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:35:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:35:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:35:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:35:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:35:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:35:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:35:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:35:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:35:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:35:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:35:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:35:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:35:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:35:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:35:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:35:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:35:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:35:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:35:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:35:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:35:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:35:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:35:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:35:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:35:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:35:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:08.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:35:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:35:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:35:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:35:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:35:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:35:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:35:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:35:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + grep -i Replaying 2023-11-24T08:35:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo "state": "Replaying" 2023-11-24T08:35:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:35:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:35:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:35:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:35:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:35:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sed -e s|[/ ]|_|g 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo cluster1-mirror-test.status 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:35:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 17 2023-11-24T08:35:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:35:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:35:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:35:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:35:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:35:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:35:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=17 2023-11-24T08:35:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:35:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:35:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:35:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:35:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:35:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:35:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:35:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:35:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:35:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:35:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:35:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:35:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 17 = 2023-11-24T08:35:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:35:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:35:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:35:11.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:35:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:35:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:35:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:35:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:35:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:35:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:35:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:35:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 17 = 2023-11-24T08:35:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:35:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:35:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:35:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:35:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:35:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:35:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:35:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:35:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:35:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:35:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:35:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 17 = 2023-11-24T08:35:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:35:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:35:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:35:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:35:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:35:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:35:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=17 2023-11-24T08:35:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 17 = 17 2023-11-24T08:35:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:35:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-11-24T08:35:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4 2023-11-24T08:35:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:35:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:35:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-11-24T08:35:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:35:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:35:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:35:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 8 2023-11-24T08:35:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:35:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:35:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:35:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:35:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:35:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:35:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:35:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:33:32 2023 2023-11-24T08:35:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:35:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:35:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:35:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:35:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:35:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:35:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:35:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snap3-clean ] 2023-11-24T08:35:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap3-clean 2023-11-24T08:35:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3-clean 2023-11-24T08:35:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:35:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:35:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:35:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:35:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:16.445 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8291952856049831) > weight (0.25) 2023-11-24T08:35:16.445 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4532470926518396) > weight (0.25) 2023-11-24T08:35:16.445 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.27894382987135546) > weight (0.25) 2023-11-24T08:35:16.445 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9846602437828833) > weight (0.25) 2023-11-24T08:35:16.446 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2023-11-24T08:35:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:35:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:35:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap4 2023-11-24T08:35:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4 2023-11-24T08:35:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:35:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:35:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:35:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:35:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:35:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:35:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:35:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap4-clean 2023-11-24T08:35:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-11-24T08:35:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:35:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4-clean 2023-11-24T08:35:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:35:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:35:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:35:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:35:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2023-11-24T08:35:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:35:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:35:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 63945728 bytes 2023-11-24T08:35:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 73100288 bytes 2023-11-24T08:35:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 142410752 bytes 2023-11-24T08:35:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 207675392 bytes 2023-11-24T08:35:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 221770752 bytes 2023-11-24T08:35:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 262132736 bytes 2023-11-24T08:35:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 362520576 bytes 2023-11-24T08:35:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 428118016 bytes 2023-11-24T08:35:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 459516928 bytes 2023-11-24T08:35:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 581782528 bytes 2023-11-24T08:35:22.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 616536064 bytes 2023-11-24T08:35:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 669282304 bytes 2023-11-24T08:35:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 785637376 bytes 2023-11-24T08:35:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 841974784 bytes 2023-11-24T08:35:23.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 862838784 bytes 2023-11-24T08:35:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 948360192 bytes 2023-11-24T08:35:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 967356416 bytes 2023-11-24T08:35:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 985360384 bytes 2023-11-24T08:35:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1023063040 bytes 2023-11-24T08:35:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1058598912 bytes 2023-11-24T08:35:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1072971776 bytes 2023-11-24T08:35:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1090999296 bytes 2023-11-24T08:35:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1111582720 bytes 2023-11-24T08:35:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1191272448 bytes 2023-11-24T08:35:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1225396224 bytes 2023-11-24T08:35:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1243020288 bytes 2023-11-24T08:35:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1305218048 bytes 2023-11-24T08:35:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1400880128 bytes 2023-11-24T08:35:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1401505792 bytes 2023-11-24T08:35:25.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1402126336 bytes 2023-11-24T08:35:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1411439616 bytes 2023-11-24T08:35:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1452212224 bytes 2023-11-24T08:35:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1465255936 bytes 2023-11-24T08:35:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1505251328 bytes 2023-11-24T08:35:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1583622144 bytes 2023-11-24T08:35:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1635495936 bytes 2023-11-24T08:35:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1657914368 bytes 2023-11-24T08:35:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1702926336 bytes 2023-11-24T08:35:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1750074368 bytes 2023-11-24T08:35:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1821643776 bytes 2023-11-24T08:35:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1929621504 bytes 2023-11-24T08:35:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1967952896 bytes 2023-11-24T08:35:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1971518464 bytes 2023-11-24T08:35:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1972141056 bytes 2023-11-24T08:35:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2025637888 bytes 2023-11-24T08:35:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2038464512 bytes 2023-11-24T08:35:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2088088576 bytes 2023-11-24T08:35:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2088708096 bytes 2023-11-24T08:35:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2110652416 bytes 2023-11-24T08:35:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2154204160 bytes 2023-11-24T08:35:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2184371200 bytes 2023-11-24T08:35:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2222749696 bytes 2023-11-24T08:35:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2271428608 bytes 2023-11-24T08:35:29.450 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.40560560057933204) > weight (0.25) 2023-11-24T08:35:29.450 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-24T08:35:29.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-24T08:35:29.451 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-11-24T08:35:29.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T08:35:29.448+0000 7f8d8170c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 39809) UID: 1000 2023-11-24T08:35:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2286730240 bytes 2023-11-24T08:35:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:35:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:35:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:35:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:35:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:35:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:35:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:35:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:35:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:35:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:35:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:35:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:35:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:35:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:35:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:35:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:35:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:35:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:35:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:35:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":12443101.87,"bytes_per_snapshot":516184064.0,"last_snapshot_bytes":533743616,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814909,"remote_snapshot_timestamp":1700814909,"replay_state":"idle"} 2023-11-24T08:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2023-11-24T08:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:35:20 2023-11-24T08:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:35:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:35:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:35:20 2023-11-24T08:35:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:35:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:35:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:35:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":12443101.87,"bytes_per_snapshot":516184064.0,"last_snapshot_bytes":533743616,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814909,"remote_snapshot_timestamp":1700814909,"replay_state":"idle"} 2023-11-24T08:35:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:35:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:35:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^up+ 2023-11-24T08:35:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:35:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:35:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2023-11-24T08:35:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:35:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:35:20 2023-11-24T08:35:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:35:20 2023-11-24T08:35:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:35:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:35:20 2023-11-24T08:35:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:35:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror.mirror_status 2023-11-24T08:35:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:35:30.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:35:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:35:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:35:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:35:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:35:20 2023-11-24T08:35:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:35:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:35:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:35:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:35:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:35:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:35:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:35:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5 2023-11-24T08:35:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap5 2023-11-24T08:35:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:35:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5 2023-11-24T08:35:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:35:36.452 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-24T08:35:36.452 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:35:36.452 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-24T08:35:36.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-24T08:35:36.453 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-24T08:35:36.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-24T08:35:36.457 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-11-24T08:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:35:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:35:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:35:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:36:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:36:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:36:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:36:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:36:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:36:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:36:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:36:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:36:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:36:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:36:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:36:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:36:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:36:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:36:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:36:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:36:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:36:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:36:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:36:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:36:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:36:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:36:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:36:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:36:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:36:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:36:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:36:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:36:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:36:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:36:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:36:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:36:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:36:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:36:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:36:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:36:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:36:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:36:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:36:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:36:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:36:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:36:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:36:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:36:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:36:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:36:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:36:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:36:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:36:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:36:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:36:01.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:36:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:36:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:36:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:36:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:36:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:36:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:36:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:36:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:36:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:36:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:36:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:36:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:36:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 21 2023-11-24T08:36:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:36:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:36:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:36:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:36:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=21 2023-11-24T08:36:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:36:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:36:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:36:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:36:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:36:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:36:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:36:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:36:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:36:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 21 = 2023-11-24T08:36:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:36:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:03.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:36:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:36:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:36:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:36:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:36:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:36:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:36:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 21 = 2023-11-24T08:36:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:36:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:36:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:36:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:36:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:36:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:36:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:36:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 21 = 2023-11-24T08:36:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:36:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:36:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=21 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 21 = 21 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:36:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:36:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-11-24T08:36:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:36:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:36:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:36:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 10 2023-11-24T08:36:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:36:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:35:16 2023 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:36:07.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:36:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:36:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snap4-clean ] 2023-11-24T08:36:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap4-clean 2023-11-24T08:36:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4-clean 2023-11-24T08:36:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:36:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:36:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:36:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:36:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:36:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap5 2023-11-24T08:36:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5 2023-11-24T08:36:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:36:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:36:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:36:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:36:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:36:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap5-clean 2023-11-24T08:36:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-11-24T08:36:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5-clean 2023-11-24T08:36:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:36:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:36:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:36:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2023-11-24T08:36:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:36:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:36:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 59867136 bytes 2023-11-24T08:36:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 116031488 bytes 2023-11-24T08:36:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 167960576 bytes 2023-11-24T08:36:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 204428288 bytes 2023-11-24T08:36:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 231583744 bytes 2023-11-24T08:36:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 256722944 bytes 2023-11-24T08:36:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 355360768 bytes 2023-11-24T08:36:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 399128576 bytes 2023-11-24T08:36:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 408834048 bytes 2023-11-24T08:36:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 433343488 bytes 2023-11-24T08:36:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 473686016 bytes 2023-11-24T08:36:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 538519552 bytes 2023-11-24T08:36:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 571774976 bytes 2023-11-24T08:36:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 611490816 bytes 2023-11-24T08:36:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 612108288 bytes 2023-11-24T08:36:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 663300096 bytes 2023-11-24T08:36:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 679165952 bytes 2023-11-24T08:36:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 722501632 bytes 2023-11-24T08:36:16.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 764659712 bytes 2023-11-24T08:36:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 781760512 bytes 2023-11-24T08:36:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 786946048 bytes 2023-11-24T08:36:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 828881920 bytes 2023-11-24T08:36:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 881170432 bytes 2023-11-24T08:36:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 945967104 bytes 2023-11-24T08:36:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1052296192 bytes 2023-11-24T08:36:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1143246848 bytes 2023-11-24T08:36:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1143847936 bytes 2023-11-24T08:36:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1170562048 bytes 2023-11-24T08:36:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1205026816 bytes 2023-11-24T08:36:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1261912064 bytes 2023-11-24T08:36:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1282012160 bytes 2023-11-24T08:36:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1302867968 bytes 2023-11-24T08:36:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1364058112 bytes 2023-11-24T08:36:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:36:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:36:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:36:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:36:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:36:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:36:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:36:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:36:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:36:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:36:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:36:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:36:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:36:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:36:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:36:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:36:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:36:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":5348352.0,"bytes_per_snapshot":516184064.0,"last_snapshot_bytes":533743616,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814909,"remote_snapshot_timestamp":1700814909,"replay_state":"idle"} 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:35:50 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:36:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:36:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:35:50 2023-11-24T08:36:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:36:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:36:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:36:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":5348352.0,"bytes_per_snapshot":516184064.0,"last_snapshot_bytes":533743616,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814909,"remote_snapshot_timestamp":1700814909,"replay_state":"idle"} 2023-11-24T08:36:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:36:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:36:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^up+ 2023-11-24T08:36:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:36:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:36:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2023-11-24T08:36:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:36:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:35:50 2023-11-24T08:36:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:35:50 2023-11-24T08:36:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:36:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:35:50 2023-11-24T08:36:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:36:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror.mirror_status 2023-11-24T08:36:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:36:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:36:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:36:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:36:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:36:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:36:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:35:50 2023-11-24T08:36:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:36:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:36:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:36:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:36:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:36:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6 2023-11-24T08:36:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap6 2023-11-24T08:36:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6 2023-11-24T08:36:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:36:40.447 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3588214425864673) > weight (0.25) 2023-11-24T08:36:40.447 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.28164500856498087) > weight (0.25) 2023-11-24T08:36:40.447 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9042131600168913) > weight (0.25) 2023-11-24T08:36:40.447 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8814801012692766) > weight (0.25) 2023-11-24T08:36:40.447 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-11-24T08:36:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:36:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:36:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:36:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:36:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:36:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:36:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:36:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:36:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:36:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:36:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:36:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:36:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:36:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:36:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:36:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i state 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:36:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:36:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:36:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:36:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:36:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:36:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:36:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:36:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:36:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:36:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:36:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:36:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:36:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:36:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:36:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:36:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:36:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:36:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:36:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:36:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:36:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:36:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:36:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:36:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:36:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:36:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:36:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:36:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:36:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:36:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:36:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:36:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:36:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:36:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:36:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:36:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:36:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:36:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:36:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:36:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:36:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:36:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:36:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:36:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:36:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:36:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:36:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:36:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:36:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:36:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:36:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:36:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:36:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:36:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:36:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:36:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:36:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:36:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:36:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:36:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:36:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:36:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:36:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:36:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:36:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:36:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:36:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:36:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:36:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:36:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:36:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:36:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:36:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:36:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 25 2023-11-24T08:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:36:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=25 2023-11-24T08:36:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:36:52.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:36:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:36:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:36:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:36:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:36:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:36:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:36:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:36:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 25 = 2023-11-24T08:36:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:36:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:36:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:36:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:36:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:36:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:36:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:36:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:36:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 25 = 2023-11-24T08:36:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:36:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:36:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:36:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:36:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:36:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:36:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:36:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:36:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 25 = 2023-11-24T08:36:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:36:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:36:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:36:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:36:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:36:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=25 2023-11-24T08:36:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 25 = 25 2023-11-24T08:36:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-11-24T08:36:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:36:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6 2023-11-24T08:36:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:36:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:36:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-11-24T08:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 12 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:35:16 2023 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:36:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:36:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:36:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snap5-clean ] 2023-11-24T08:36:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap5-clean 2023-11-24T08:36:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5-clean 2023-11-24T08:36:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:36:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:36:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:36:57.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:36:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:36:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:36:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap6 2023-11-24T08:36:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:36:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:36:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6 2023-11-24T08:36:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:36:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:36:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:36:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:37:00.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:37:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:37:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap6-clean 2023-11-24T08:37:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-11-24T08:37:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:37:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6-clean 2023-11-24T08:37:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:37:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:37:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:37:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:37:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2023-11-24T08:37:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:37:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:37:03.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 72307712 bytes 2023-11-24T08:37:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 123237376 bytes 2023-11-24T08:37:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 174958592 bytes 2023-11-24T08:37:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 289467392 bytes 2023-11-24T08:37:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 335208448 bytes 2023-11-24T08:37:04.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 391448576 bytes 2023-11-24T08:37:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 410643456 bytes 2023-11-24T08:37:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 436483072 bytes 2023-11-24T08:37:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 468329472 bytes 2023-11-24T08:37:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 526107648 bytes 2023-11-24T08:37:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 537107456 bytes 2023-11-24T08:37:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 568009728 bytes 2023-11-24T08:37:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 628582400 bytes 2023-11-24T08:37:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 671534080 bytes 2023-11-24T08:37:05.458 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4308148764802685) > weight (0.25) 2023-11-24T08:37:05.458 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5827323158241876) > weight (0.25) 2023-11-24T08:37:05.458 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7381545767200021) > weight (0.25) 2023-11-24T08:37:05.459 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6047075167134601) > weight (0.25) 2023-11-24T08:37:05.459 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2023-11-24T08:37:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 736743424 bytes 2023-11-24T08:37:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 786994176 bytes 2023-11-24T08:37:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 810560512 bytes 2023-11-24T08:37:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 817672192 bytes 2023-11-24T08:37:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 903272448 bytes 2023-11-24T08:37:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 999838720 bytes 2023-11-24T08:37:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1061970944 bytes 2023-11-24T08:37:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1110183936 bytes 2023-11-24T08:37:07.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1206156288 bytes 2023-11-24T08:37:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1274952704 bytes 2023-11-24T08:37:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1299514368 bytes 2023-11-24T08:37:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1334524928 bytes 2023-11-24T08:37:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1350595584 bytes 2023-11-24T08:37:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1373737984 bytes 2023-11-24T08:37:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1396209664 bytes 2023-11-24T08:37:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1451421696 bytes 2023-11-24T08:37:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1507738624 bytes 2023-11-24T08:37:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1554824192 bytes 2023-11-24T08:37:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1570837504 bytes 2023-11-24T08:37:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1607570432 bytes 2023-11-24T08:37:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1718888448 bytes 2023-11-24T08:37:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1737990144 bytes 2023-11-24T08:37:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1780302848 bytes 2023-11-24T08:37:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1800211456 bytes 2023-11-24T08:37:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1834659840 bytes 2023-11-24T08:37:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1849004032 bytes 2023-11-24T08:37:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1904587776 bytes 2023-11-24T08:37:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1946607616 bytes 2023-11-24T08:37:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2016723968 bytes 2023-11-24T08:37:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2076992512 bytes 2023-11-24T08:37:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2099443712 bytes 2023-11-24T08:37:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2156890112 bytes 2023-11-24T08:37:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2213494784 bytes 2023-11-24T08:37:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2293614592 bytes 2023-11-24T08:37:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2327170048 bytes 2023-11-24T08:37:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2375245824 bytes 2023-11-24T08:37:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2498723840 bytes 2023-11-24T08:37:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2552775680 bytes 2023-11-24T08:37:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2685460480 bytes 2023-11-24T08:37:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2708168704 bytes 2023-11-24T08:37:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2763023360 bytes 2023-11-24T08:37:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2837143552 bytes 2023-11-24T08:37:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2883931136 bytes 2023-11-24T08:37:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2927998976 bytes 2023-11-24T08:37:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2944148480 bytes 2023-11-24T08:37:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2966993920 bytes 2023-11-24T08:37:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 3039942656 bytes 2023-11-24T08:37:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 3074538496 bytes 2023-11-24T08:37:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 3137230848 bytes 2023-11-24T08:37:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 3190494208 bytes 2023-11-24T08:37:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 3212302336 bytes 2023-11-24T08:37:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3243103232 bytes 2023-11-24T08:37:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3297032192 bytes 2023-11-24T08:37:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3320579072 bytes 2023-11-24T08:37:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3343086592 bytes 2023-11-24T08:37:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3389741056 bytes 2023-11-24T08:37:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3435551744 bytes 2023-11-24T08:37:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:37:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:37:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:37:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:37:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:37:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:37:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:37:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:37:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:37:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:37:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:37:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:37:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:37:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:37:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:37:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:37:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:37:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:37:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:37:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:37:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:37:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527866880.0,"last_snapshot_bytes":521990144,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814961,"remote_snapshot_timestamp":1700814961,"replay_state":"idle"} 2023-11-24T08:37:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2023-11-24T08:37:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:36:50 2023-11-24T08:37:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:37:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:37:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:37:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:37:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:36:50 2023-11-24T08:37:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:37:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:37:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:37:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":527866880.0,"last_snapshot_bytes":521990144,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700814961,"remote_snapshot_timestamp":1700814961,"replay_state":"idle"} 2023-11-24T08:37:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:37:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:37:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^up+ 2023-11-24T08:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:37:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2023-11-24T08:37:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:37:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:36:50 2023-11-24T08:37:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:36:50 2023-11-24T08:37:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:37:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:36:50 2023-11-24T08:37:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:37:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror.mirror_status 2023-11-24T08:37:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:37:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:37:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:37:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:37:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:37:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:37:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:36:50 2023-11-24T08:37:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:37:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:37:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:37:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:37:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:37:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7 2023-11-24T08:37:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap7 2023-11-24T08:37:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:37:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7 2023-11-24T08:37:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:37:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:37:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:37:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:37:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:37:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:37:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:37:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:37:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:37:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:37:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ cluster=cluster1 2023-11-24T08:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:37:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:37:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:37:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:37:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:37:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:37:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:37:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:37:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:37:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:37:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:37:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:37:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:37:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:37:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:37:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:37:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:37:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:37:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:37:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:37:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:37:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:37:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:37:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:37:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:37:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:37:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:37:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:37:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:37:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:37:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:37:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:37:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:37:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:37:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:37:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:37:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:37:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:37:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:37:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:37:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:37:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:37:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:37:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:37:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:37:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:37:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:37:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:37:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:37:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:37:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:37:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:37:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:37:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:37:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:37:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:37:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:37:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:37:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:37:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:37:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:37:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:37:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:37:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:37:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:37:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:37:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:37:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:37:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:37:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:37:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:37:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:37:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:37:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:37:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:37:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:37:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:37:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:37:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:37:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:37:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:37:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:37:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:37:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:37:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:37:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:37:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:37:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:37:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:37:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 29 2023-11-24T08:37:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:37:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:37:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:37:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:37:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:37:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:37:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=29 2023-11-24T08:37:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:37:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:37:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:37:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:37:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:37:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:37:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:37:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:37:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:37:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:37:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:37:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:37:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 29 = 2023-11-24T08:37:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:37:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:37:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:37:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:37:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:37:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:37:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:37:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:37:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:37:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:37:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:37:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 29 = 2023-11-24T08:37:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:37:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:37:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:37:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:37:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:37:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:37:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:37:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:37:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:37:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:37:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:37:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 29 = 2023-11-24T08:37:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:37:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:37:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:37:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:37:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:37:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:37:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=29 2023-11-24T08:37:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 29 = 29 2023-11-24T08:37:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-11-24T08:37:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:37:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7 2023-11-24T08:37:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:37:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:37:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-11-24T08:37:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:37:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:37:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 14 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:36:58 2023 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:37:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:37:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:37:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:37:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snap6-clean ] 2023-11-24T08:37:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap6-clean 2023-11-24T08:37:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6-clean 2023-11-24T08:37:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:37:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:37:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:37:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:37:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:55.448 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-11-24T08:37:55.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-11-24T08:37:55.449 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-11-24T08:37:55.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T08:37:55.447+0000 7f6e58395640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 39816) UID: 1000 2023-11-24T08:37:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:37:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap7 2023-11-24T08:37:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7 2023-11-24T08:37:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:37:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:37:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:56.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:37:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:37:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:37:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:37:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:37:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap7-clean 2023-11-24T08:37:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-11-24T08:37:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:37:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7-clean 2023-11-24T08:37:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:37:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:37:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:37:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:37:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:37:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:37:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T08:37:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:37:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:38:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 52733952 bytes 2023-11-24T08:38:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 53837824 bytes 2023-11-24T08:38:00.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 197809152 bytes 2023-11-24T08:38:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 208309248 bytes 2023-11-24T08:38:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 213422080 bytes 2023-11-24T08:38:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 230504448 bytes 2023-11-24T08:38:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 243531776 bytes 2023-11-24T08:38:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 313677824 bytes 2023-11-24T08:38:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 337788928 bytes 2023-11-24T08:38:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 435512320 bytes 2023-11-24T08:38:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 483717120 bytes 2023-11-24T08:38:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 550511616 bytes 2023-11-24T08:38:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 635248640 bytes 2023-11-24T08:38:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 674546688 bytes 2023-11-24T08:38:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 689800192 bytes 2023-11-24T08:38:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 736879616 bytes 2023-11-24T08:38:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 769381376 bytes 2023-11-24T08:38:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 784217088 bytes 2023-11-24T08:38:03.450 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-11-24T08:38:03.450 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:38:03.451 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-11-24T08:38:03.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-11-24T08:38:03.451 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-11-24T08:38:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-11-24T08:38:03.453 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-11-24T08:38:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 885633024 bytes 2023-11-24T08:38:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 952010752 bytes 2023-11-24T08:38:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1059633152 bytes 2023-11-24T08:38:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1088326656 bytes 2023-11-24T08:38:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1124861952 bytes 2023-11-24T08:38:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1165404160 bytes 2023-11-24T08:38:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1214070784 bytes 2023-11-24T08:38:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1249670144 bytes 2023-11-24T08:38:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1257939968 bytes 2023-11-24T08:38:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1275444224 bytes 2023-11-24T08:38:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1316889600 bytes 2023-11-24T08:38:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1369477120 bytes 2023-11-24T08:38:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1408749568 bytes 2023-11-24T08:38:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1446621184 bytes 2023-11-24T08:38:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1493335040 bytes 2023-11-24T08:38:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1531650048 bytes 2023-11-24T08:38:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1572210688 bytes 2023-11-24T08:38:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1623223296 bytes 2023-11-24T08:38:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1680790528 bytes 2023-11-24T08:38:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1725473792 bytes 2023-11-24T08:38:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1758098432 bytes 2023-11-24T08:38:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1859278848 bytes 2023-11-24T08:38:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1899214848 bytes 2023-11-24T08:38:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1899823104 bytes 2023-11-24T08:38:07.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2002506752 bytes 2023-11-24T08:38:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2010884096 bytes 2023-11-24T08:38:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2058086400 bytes 2023-11-24T08:38:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2069393408 bytes 2023-11-24T08:38:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2096261120 bytes 2023-11-24T08:38:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2124894208 bytes 2023-11-24T08:38:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2193538048 bytes 2023-11-24T08:38:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2194157568 bytes 2023-11-24T08:38:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2225209344 bytes 2023-11-24T08:38:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2270968832 bytes 2023-11-24T08:38:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2295274496 bytes 2023-11-24T08:38:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2418718720 bytes 2023-11-24T08:38:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2429092864 bytes 2023-11-24T08:38:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2486123520 bytes 2023-11-24T08:38:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:38:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:38:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:38:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:38:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:38:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:38:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:38:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:38:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:38:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:38:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:38:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:38:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:38:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:38:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:38:10.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":514881024.0,"last_snapshot_bytes":507771904,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700815011,"remote_snapshot_timestamp":1700815011,"replay_state":"idle"} 2023-11-24T08:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2023-11-24T08:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:37:50 2023-11-24T08:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: down+unknown 2023-11-24T08:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: status not found 2023-11-24T08:38:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24T08:38:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:38:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:38:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:38:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":514881024.0,"last_snapshot_bytes":507771904,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700815011,"remote_snapshot_timestamp":1700815011,"replay_state":"idle"} 2023-11-24T08:38:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:38:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:38:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^up+ 2023-11-24T08:38:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:38:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:38:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2023-11-24T08:38:10.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:38:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:37:50 2023-11-24T08:38:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:37:50 2023-11-24T08:38:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:38:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:37:50 2023-11-24T08:38:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:38:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror.mirror_status 2023-11-24T08:38:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:38:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:38:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:38:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:38:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:38:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:38:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:37:50 2023-11-24T08:38:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:38:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:38:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:38:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:38:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:38:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8 2023-11-24T08:38:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap8 2023-11-24T08:38:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:38:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8 2023-11-24T08:38:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:38:31.460 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5263627484663022) > weight (0.25) 2023-11-24T08:38:31.460 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-24T08:38:31.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-24T08:38:31.461 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-24T08:38:31.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T08:38:31.462+0000 7f49d3743640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 43341) UID: 1000 2023-11-24T08:38:34.462 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-24T08:38:34.462 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:38:34.462 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-24T08:38:34.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-24T08:38:34.463 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-24T08:38:34.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-24T08:38:34.466 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2023-11-24T08:38:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:38:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:38:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:38:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:38:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:38:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:38:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:38:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:38:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:38:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:38:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:38:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:38:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ localgrep cluster=cluster1 -i 2023-11-24T08:38:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state 2023-11-24T08:38:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:38:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:38:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:38:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:38:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:38:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:38:41.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:38:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:38:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:38:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:38:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:38:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:38:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:38:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:38:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:38:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:38:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:38:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:38:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:38:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:38:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:38:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:38:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:38:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:38:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:38:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:38:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:38:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:38:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:38:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:38:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:38:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:38:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:38:42.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:38:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:38:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:38:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:38:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:38:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:38:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:38:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:38:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:38:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:38:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:38:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:38:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:38:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:38:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:38:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:38:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:38:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:38:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:38:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:38:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:38:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:38:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:38:42.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:38:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:38:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:38:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:38:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:38:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:38:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:38:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:38:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:38:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:38:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 33 2023-11-24T08:38:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:38:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:38:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:38:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:38:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:38:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:38:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=33 2023-11-24T08:38:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:38:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:38:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:38:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:38:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:38:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:38:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:38:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:38:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:38:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:38:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:38:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:38:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 33 = 2023-11-24T08:38:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:38:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:38:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:38:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:38:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:38:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:38:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:38:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:38:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:38:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:38:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:38:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 33 = 2023-11-24T08:38:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:38:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:38:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:38:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:38:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:38:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:38:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:38:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:38:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:38:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:38:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:38:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 33 = 2023-11-24T08:38:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:38:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:38:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:38:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:38:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:38:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=33 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 33 = 33 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:38:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-11-24T08:38:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:38:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:38:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:38:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 16 2023-11-24T08:38:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:38:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:38:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:38:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:38:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:38:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:38:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:38:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:36:58 2023 2023-11-24T08:38:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:38:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:38:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:38:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:38:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:38:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:38:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snap7-clean ] 2023-11-24T08:38:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap7-clean 2023-11-24T08:38:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7-clean 2023-11-24T08:38:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:38:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:38:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:38:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:38:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:38:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap8 2023-11-24T08:38:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8 2023-11-24T08:38:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:38:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:38:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:38:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:38:51.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:38:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap8-clean 2023-11-24T08:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-11-24T08:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8-clean 2023-11-24T08:38:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:38:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:38:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:38:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:38:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2023-11-24T08:38:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:38:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:38:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 36708352 bytes 2023-11-24T08:38:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 101931008 bytes 2023-11-24T08:38:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 173096960 bytes 2023-11-24T08:38:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 173725696 bytes 2023-11-24T08:38:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 174363648 bytes 2023-11-24T08:38:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 227083264 bytes 2023-11-24T08:38:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 264154112 bytes 2023-11-24T08:38:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 304178176 bytes 2023-11-24T08:38:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 349074432 bytes 2023-11-24T08:38:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 402021376 bytes 2023-11-24T08:38:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 402977792 bytes 2023-11-24T08:38:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 452854784 bytes 2023-11-24T08:38:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 468076544 bytes 2023-11-24T08:38:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 502529024 bytes 2023-11-24T08:38:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 527938560 bytes 2023-11-24T08:38:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 586371072 bytes 2023-11-24T08:38:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 608680960 bytes 2023-11-24T08:38:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 631516160 bytes 2023-11-24T08:38:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 669214720 bytes 2023-11-24T08:38:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 695983104 bytes 2023-11-24T08:38:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 729849856 bytes 2023-11-24T08:38:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 762469376 bytes 2023-11-24T08:38:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 789666816 bytes 2023-11-24T08:38:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 869242880 bytes 2023-11-24T08:38:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 878842880 bytes 2023-11-24T08:38:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 909679616 bytes 2023-11-24T08:38:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1004198912 bytes 2023-11-24T08:38:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1042920448 bytes 2023-11-24T08:38:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1088533504 bytes 2023-11-24T08:38:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1164454912 bytes 2023-11-24T08:38:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1204808704 bytes 2023-11-24T08:38:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1262996480 bytes 2023-11-24T08:38:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1316640768 bytes 2023-11-24T08:38:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1334320128 bytes 2023-11-24T08:38:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1381624832 bytes 2023-11-24T08:38:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:38:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:38:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:38:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:38:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:38:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:38:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:38:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:38:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:38:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:38:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:38:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:39:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:39:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:39:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:39:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:39:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:39:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:39:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:39:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:39:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:39:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:39:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:39:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":13683575.47,"bytes_per_snapshot":534864384.0,"last_snapshot_bytes":535427072,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700815122,"remote_snapshot_timestamp":1700815122,"replay_state":"idle"} 2023-11-24T08:39:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2023-11-24T08:39:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:38:50 2023-11-24T08:39:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:39:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:39:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:39:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:39:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:38:50 2023-11-24T08:39:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:39:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":13683575.47,"bytes_per_snapshot":534864384.0,"last_snapshot_bytes":535427072,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700815122,"remote_snapshot_timestamp":1700815122,"replay_state":"idle"} 2023-11-24T08:39:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:39:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo+ up+replaying 2023-11-24T08:39:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep ^up+ 2023-11-24T08:39:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:39:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:38:50 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:38:50 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:38:50 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:39:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror.mirror_status 2023-11-24T08:39:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:39:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:39:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:39:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:39:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:39:00.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:39:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:38:50 2023-11-24T08:39:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:39:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:39:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:39:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:39:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:39:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9 2023-11-24T08:39:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap9 2023-11-24T08:39:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:39:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9 2023-11-24T08:39:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:39:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:39:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:39:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:39:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:39:31.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:39:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:39:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:39:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:39:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:39:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local loop_instance 2023-11-24T08:39:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:39:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:39:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:39:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:39:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:39:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:39:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:39:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:39:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:39:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:39:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:39:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:39:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:39:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:39:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:39:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:39:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:39:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:39:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:39:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:39:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:39:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:39:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:39:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:39:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:39:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:39:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:39:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:39:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:39:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:39:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:39:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:39:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:39:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:39:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:39:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:39:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:39:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:39:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:39:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:39:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:39:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:39:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:39:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:39:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:39:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:39:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:39:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:39:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:39:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:39:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:39:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:39:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:39:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:39:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:39:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:39:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:39:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:39:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:39:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:39:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:39:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:39:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:39:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:39:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:39:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:39:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:39:31.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:39:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:39:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:39:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:39:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:39:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:39:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:39:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:39:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:39:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:39:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:39:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:39:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:39:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:39:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:39:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:39:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:39:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:39:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:39:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:39:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:39:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:39:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:39:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:39:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:39:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:39:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:39:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:39:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:39:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:39:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:39:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:39:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:39:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:39:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:39:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:39:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:39:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:39:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:39:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:39:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:39:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:39:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:39:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:39:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:39:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:39:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:39:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:39:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:39:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:39:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:39:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 37 2023-11-24T08:39:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:39:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:39:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:39:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:39:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:39:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:39:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=37 2023-11-24T08:39:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:39:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:39:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:39:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:39:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:39:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:39:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:39:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:39:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:39:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:39:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 37 = 2023-11-24T08:39:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:39:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:39:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:39:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:39:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:39:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:39:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:39:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:39:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:39:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:39:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:39:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 37 = 2023-11-24T08:39:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:39:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:39:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:39:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:39:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:39:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:39:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:39:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:39:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:39:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:39:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:39:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 37 = 2023-11-24T08:39:35.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:39:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:39:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:39:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:39:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:39:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:39:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=37 2023-11-24T08:39:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 37 = 37 2023-11-24T08:39:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-11-24T08:39:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9 2023-11-24T08:39:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:39:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:39:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 18 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:39:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:39:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:39:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:38:49 2023 2023-11-24T08:39:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:39:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:39:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:39:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:39:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:39:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:39:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snap8-clean ] 2023-11-24T08:39:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap8-clean 2023-11-24T08:39:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8-clean 2023-11-24T08:39:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:39:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:39:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:39:38.454 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7516339460458448) > weight (0.25) 2023-11-24T08:39:38.455 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-24T08:39:38.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-24T08:39:38.455 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-11-24T08:39:38.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T08:39:38.458+0000 7ff743dd9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 41902) UID: 1000 2023-11-24T08:39:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:39:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:39:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap9 2023-11-24T08:39:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9 2023-11-24T08:39:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:39:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:39:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:40.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:39:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:39:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:39:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:39:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap9-clean 2023-11-24T08:39:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-11-24T08:39:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:39:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9-clean 2023-11-24T08:39:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:39:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror test 2023-11-24T08:39:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:39:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:39:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2023-11-24T08:39:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:39:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-24T08:39:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 59293696 bytes 2023-11-24T08:39:44.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 97922048 bytes 2023-11-24T08:39:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 170585088 bytes 2023-11-24T08:39:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 264605696 bytes 2023-11-24T08:39:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 330652672 bytes 2023-11-24T08:39:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 376111104 bytes 2023-11-24T08:39:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 394222592 bytes 2023-11-24T08:39:45.456 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-24T08:39:45.457 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:39:45.457 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-24T08:39:45.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-24T08:39:45.457 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-24T08:39:45.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-24T08:39:45.459 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-11-24T08:39:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 449511424 bytes 2023-11-24T08:39:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 460161024 bytes 2023-11-24T08:39:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 513803264 bytes 2023-11-24T08:39:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 638710784 bytes 2023-11-24T08:39:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 673971200 bytes 2023-11-24T08:39:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 732138496 bytes 2023-11-24T08:39:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 757521408 bytes 2023-11-24T08:39:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 805320704 bytes 2023-11-24T08:39:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 842631168 bytes 2023-11-24T08:39:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 881431552 bytes 2023-11-24T08:39:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 961150976 bytes 2023-11-24T08:39:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1005050880 bytes 2023-11-24T08:39:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1062219776 bytes 2023-11-24T08:39:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1122135040 bytes 2023-11-24T08:39:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1122749440 bytes 2023-11-24T08:39:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1174672384 bytes 2023-11-24T08:39:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1237047296 bytes 2023-11-24T08:39:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1256573952 bytes 2023-11-24T08:39:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1279025152 bytes 2023-11-24T08:39:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:39:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:39:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:39:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:39:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:39:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:39:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:39:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:39:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-24T08:39:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:39:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local description_pattern= 2023-11-24T08:39:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local service_pattern= 2023-11-24T08:39:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-11-24T08:39:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.mirror_status 2023-11-24T08:39:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:39:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-11-24T08:39:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:test: 2023-11-24T08:39:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: global_id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:39:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+replaying 2023-11-24T08:39:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: replaying, {"bytes_per_second":17847569.07,"bytes_per_snapshot":534864384.0,"last_snapshot_bytes":535427072,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700815122,"remote_snapshot_timestamp":1700815122,"replay_state":"idle"} 2023-11-24T08:39:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: service: mirror.2 on smithi164 2023-11-24T08:39:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:39:20 2023-11-24T08:39:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: peer_sites: 2023-11-24T08:39:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: name: cluster2 2023-11-24T08:39:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state: up+stopped 2023-11-24T08:39:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: description: local image is primary 2023-11-24T08:39:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: last_update: 2023-11-24 08:39:20 2023-11-24T08:39:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: state: up+replaying 2023-11-24T08:39:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^ description: .* /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: description: replaying, {"bytes_per_second":17847569.07,"bytes_per_snapshot":534864384.0,"last_snapshot_bytes":535427072,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1700815122,"remote_snapshot_timestamp":1700815122,"replay_state":"idle"} 2023-11-24T08:39:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:39:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:39:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep ^up+ 2023-11-24T08:39:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:39:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep service: *mirror..* on /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: service: mirror.2 on smithi164 2023-11-24T08:39:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.mirror_status 2023-11-24T08:39:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update=2023-11-24 08:39:20 2023-11-24T08:39:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-11-24 08:39:20 2023-11-24T08:39:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:39:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state_pattern=up+replaying 2023-11-24T08:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local prev_last_update=2023-11-24 08:39:20 2023-11-24T08:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror.mirror_status 2023-11-24T08:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror.mirror_status 2023-11-24T08:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror.mirror_status 2023-11-24T08:39:49.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-24T08:39:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:39:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_update state 2023-11-24T08:39:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-11-24T08:39:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ last_update=2023-11-24 08:39:20 2023-11-24T08:39:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-11-24T08:39:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=up+replaying 2023-11-24T08:39:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo up+replaying 2023-11-24T08:39:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep up+replaying 2023-11-24T08:39:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:up+replaying 2023-11-24T08:39:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:39:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10 2023-11-24T08:39:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap10 2023-11-24T08:39:50.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:39:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:39:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:39:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10 2023-11-24T08:39:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:40:07.467 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6168422840164097) > weight (0.25) 2023-11-24T08:40:07.467 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4299818012773573) > weight (0.25) 2023-11-24T08:40:07.467 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-11-24T08:40:07.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-11-24T08:40:07.468 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-24T08:40:07.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:40:07.470+0000 7ff445ce2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 39811) UID: 1000 2023-11-24T08:40:09.469 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-11-24T08:40:09.469 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:40:09.470 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-11-24T08:40:09.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-11-24T08:40:09.470 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-11-24T08:40:09.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-11-24T08:40:09.473 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-11-24T08:40:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:40:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:40:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:40:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:40:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:40:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:40:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:40:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:40:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:40:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:40:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:40:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:40:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:40:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:40:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:40:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:40:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:40:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:40:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:20.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:40:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:40:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:40:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:40:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:40:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:40:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:40:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:40:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:40:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:40:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:40:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:40:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:40:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:21.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:40:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:40:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:40:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:40:22.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:40:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:40:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:40:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:40:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:40:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:40:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:40:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:40:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:40:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:40:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:40:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:40:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:40:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:40:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:40:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:40:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:40:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:40:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:40:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:40:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:40:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:40:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:40:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:40:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:40:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:40:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:40:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:40:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:40:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:40:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:40:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:40:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:40:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:40:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:40:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:25.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:40:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:40:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:40:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:40:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:40:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:40:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T08:40:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:40:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:40:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:29.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:40:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:40:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:40:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:40:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:40:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:40:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:40:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:40:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:40:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:40:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:29.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:40:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:40:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:40:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:40:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:40:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:40:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:40:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:40:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:40:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:40:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:40:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:40:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:40:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:40:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:40:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:40:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:40:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:40:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:40:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:40:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:40:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:40:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:40:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:40:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:40:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:40:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:40:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:40:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:40:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:40:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:40:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:40:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:40:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:40:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:40:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:40:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:46.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:40:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:40:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:40:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:40:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:40:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:40:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:40:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:40:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:40:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:40:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:40:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:40:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:40:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:40:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:40:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:40:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:40:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:40:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:40:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:40:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:40:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:40:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:40:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:40:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:40:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:40:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:40:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:40:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:40:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 41 2023-11-24T08:40:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:40:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:40:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:40:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:40:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:40:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:40:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=41 2023-11-24T08:40:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:40:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:40:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:40:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:40:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:40:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:40:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:40:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:40:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:40:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:40:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:40:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:40:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 41 = 2023-11-24T08:40:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:40:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:40:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:40:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:40:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:40:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:40:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:40:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:40:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:40:49.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:40:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:40:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 41 = 2023-11-24T08:40:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:40:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:40:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:40:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:40:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:40:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:40:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:40:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:40:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:40:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:40:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:40:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 41 = 2023-11-24T08:40:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:40:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:40:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:40:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:40:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:40:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:40:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=41 2023-11-24T08:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 41 = 41 2023-11-24T08:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-11-24T08:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10 2023-11-24T08:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:40:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:40:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-11-24T08:40:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 20 2023-11-24T08:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:40:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:38:49 2023 2023-11-24T08:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:40:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:40:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:40:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snap9-clean ] 2023-11-24T08:40:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap9-clean 2023-11-24T08:40:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9-clean 2023-11-24T08:40:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:40:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:40:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:40:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:40:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:40:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:40:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap10 2023-11-24T08:40:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10 2023-11-24T08:40:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:40:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:40:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:40:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:40:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:40:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:40:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:40:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ clean_snap_name=snap10-clean 2023-11-24T08:40:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-11-24T08:40:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:40:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10-clean 2023-11-24T08:40:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:40:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:40:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:40:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-24T08:40:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:40:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:40:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:40:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:40:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:40:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-24T08:40:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:40:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + shift 2023-11-24T08:40:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1grep 2023-11-24T08:40:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -i state 2023-11-24T08:40:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:40:59.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:40:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:40:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:40:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:40:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:40:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:40:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:40:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:40:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-24T08:40:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:40:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:40:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-24T08:40:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:40:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:40:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:40:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-24T08:40:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:40:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:40:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:40:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:40:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:40:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:40:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:40:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:40:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-24T08:41:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-24T08:41:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:41:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:41:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-24T08:41:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:41:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:41:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:41:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:41:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:41:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:41:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:41:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:41:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-24T08:41:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:41:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:41:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:41:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:41:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:41:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:41:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-24T08:41:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:41:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-test.status 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-test.status 2023-11-24T08:41:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:41:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:41:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-test.status 2023-11-24T08:41:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-test.status 2023-11-24T08:41:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:41:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:41:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-24T08:41:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-24T08:41:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 44 2023-11-24T08:41:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:41:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.status 2023-11-24T08:41:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-24T08:41:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:41:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:41:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=44 2023-11-24T08:41:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:41:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:41:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:41:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:41:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.status 2023-11-24T08:41:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-24T08:41:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:41:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=44 2023-11-24T08:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 44 = 44 2023-11-24T08:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-11-24T08:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10-clean 2023-11-24T08:41:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:41:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:41:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-11-24T08:41:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'test': 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 21 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10a5f8dc243e 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10a5f8dc243e 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:40:54 2023 2023-11-24T08:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:31:46 2023 2023-11-24T08:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 353c2fb6-3fa9-489c-a5d0-022230ee7a07 2023-11-24T08:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:41:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 10 2023-11-24T08:41:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1 2023-11-24T08:41:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap1 2023-11-24T08:41:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1 2023-11-24T08:41:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:04.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1-clean 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap1-clean 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap1-clean 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2 2023-11-24T08:41:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap2 2023-11-24T08:41:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2 2023-11-24T08:41:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:08.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2-clean 2023-11-24T08:41:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap2-clean 2023-11-24T08:41:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap2-clean 2023-11-24T08:41:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3 2023-11-24T08:41:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap3 2023-11-24T08:41:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3 2023-11-24T08:41:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:13.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3-clean 2023-11-24T08:41:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap3-clean 2023-11-24T08:41:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap3-clean 2023-11-24T08:41:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4 2023-11-24T08:41:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap4 2023-11-24T08:41:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4 2023-11-24T08:41:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4-clean 2023-11-24T08:41:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap4-clean 2023-11-24T08:41:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap4-clean 2023-11-24T08:41:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5 2023-11-24T08:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap5 2023-11-24T08:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5 2023-11-24T08:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5-clean 2023-11-24T08:41:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap5-clean 2023-11-24T08:41:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap5-clean 2023-11-24T08:41:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6 2023-11-24T08:41:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap6 2023-11-24T08:41:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6 2023-11-24T08:41:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6-clean 2023-11-24T08:41:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap6-clean 2023-11-24T08:41:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap6-clean 2023-11-24T08:41:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:30.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5109246912003238) > weight (0.25) 2023-11-24T08:41:30.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8232141604883897) > weight (0.25) 2023-11-24T08:41:30.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.780331782102115) > weight (0.25) 2023-11-24T08:41:30.461 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-24T08:41:30.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-24T08:41:30.462 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-24T08:41:30.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T08:41:30.461+0000 7fc404470640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 39841) UID: 1000 2023-11-24T08:41:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7 2023-11-24T08:41:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap7 2023-11-24T08:41:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7 2023-11-24T08:41:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:32.462 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-24T08:41:32.463 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:41:32.463 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-24T08:41:32.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-24T08:41:32.463 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-24T08:41:32.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-24T08:41:32.466 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-11-24T08:41:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7-clean 2023-11-24T08:41:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap7-clean 2023-11-24T08:41:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap7-clean 2023-11-24T08:41:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8 2023-11-24T08:41:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap8 2023-11-24T08:41:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8 2023-11-24T08:41:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8-clean 2023-11-24T08:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap8-clean 2023-11-24T08:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap8-clean 2023-11-24T08:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:39.474 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-24T08:41:39.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-24T08:41:39.475 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-11-24T08:41:39.475 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-24T08:41:39.475 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:41:39.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T08:41:39.476+0000 7f6b27f39640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41810) UID: 1000 2023-11-24T08:41:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9 2023-11-24T08:41:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap9 2023-11-24T08:41:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9 2023-11-24T08:41:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9-clean 2023-11-24T08:41:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap9-clean 2023-11-24T08:41:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap9-clean 2023-11-24T08:41:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap10 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:45.576 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-24T08:41:45.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-24T08:41:45.576 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-24T08:41:45.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-24T08:41:45.581 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2023-11-24T08:41:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10-clean 2023-11-24T08:41:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror test snap10-clean 2023-11-24T08:41:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap10-clean 2023-11-24T08:41:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:41:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export 2023-11-24T08:41:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-24T08:41:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-test.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-test.export 2023-11-24T08:41:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:41:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 10 2023-11-24T08:41:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1 2023-11-24T08:41:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-11-24T08:41:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap1 2023-11-24T08:41:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-11-24T08:41:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap1-clean 2023-11-24T08:41:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-11-24T08:41:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap1-clean 2023-11-24T08:41:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-11-24T08:41:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2 2023-11-24T08:41:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-11-24T08:41:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap2 2023-11-24T08:41:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-11-24T08:41:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap2-clean 2023-11-24T08:41:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-11-24T08:41:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap2-clean 2023-11-24T08:41:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-11-24T08:41:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3 2023-11-24T08:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-11-24T08:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap3 2023-11-24T08:41:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-11-24T08:41:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap3-clean 2023-11-24T08:41:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-11-24T08:41:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap3-clean 2023-11-24T08:41:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-11-24T08:41:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4 2023-11-24T08:41:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-11-24T08:41:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap4 2023-11-24T08:41:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-11-24T08:41:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap4-clean 2023-11-24T08:41:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-11-24T08:41:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap4-clean 2023-11-24T08:41:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-11-24T08:41:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5 2023-11-24T08:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-11-24T08:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap5 2023-11-24T08:41:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-11-24T08:41:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap5-clean 2023-11-24T08:41:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-11-24T08:41:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap5-clean 2023-11-24T08:41:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-11-24T08:41:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6 2023-11-24T08:41:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-11-24T08:41:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap6 2023-11-24T08:41:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-11-24T08:41:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap6-clean 2023-11-24T08:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-11-24T08:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:41:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:41:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:41:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap6-clean 2023-11-24T08:41:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-11-24T08:42:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7 2023-11-24T08:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-11-24T08:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap7 2023-11-24T08:42:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-11-24T08:42:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:42:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap7-clean 2023-11-24T08:42:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-11-24T08:42:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap7-clean 2023-11-24T08:42:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-11-24T08:42:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:42:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8 2023-11-24T08:42:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-11-24T08:42:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap8 2023-11-24T08:42:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-11-24T08:42:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:42:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap8-clean 2023-11-24T08:42:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-11-24T08:42:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap8-clean 2023-11-24T08:42:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-11-24T08:42:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:42:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9 2023-11-24T08:42:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-11-24T08:42:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap9 2023-11-24T08:42:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-11-24T08:42:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:42:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap9-clean 2023-11-24T08:42:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-11-24T08:42:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap9-clean 2023-11-24T08:42:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-11-24T08:42:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:42:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10 2023-11-24T08:42:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-11-24T08:42:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap10 2023-11-24T08:42:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-11-24T08:42:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:42:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap10-clean 2023-11-24T08:42:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-11-24T08:42:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap10-clean 2023-11-24T08:42:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-11-24T08:42:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T08:42:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror test 2023-11-24T08:42:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:42:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror test 2023-11-24T08:42:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-11-24T08:42:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-11-24T08:42:09.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T08:42:09.196+0000 7f564d07d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55fcbc12a820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T08:42:09.198 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T08:42:09.196+0000 7f5653089640 -1 rbd::mirror::ImageReplayer: 0x55fcbb690280 [3/353c2fb6-3fa9-489c-a5d0-022230ee7a07] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T08:42:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T08:42:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-11-24T08:42:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:42:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T08:42:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T08:42:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:42:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T08:42:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror test 2023-11-24T08:42:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:42:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-11-24T08:42:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T08:42:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image test: (2) No such file or directory 2023-11-24T08:42:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T08:42:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T08:42:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror test deleted 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror test 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=test 2023-11-24T08:42:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-11-24T08:42:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T08:42:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image test: (2) No such file or directory 2023-11-24T08:42:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T08:42:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T08:42:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T08:42:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog TEST: create many images 2023-11-24T08:42:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.tyy3Nh9UuT/rbd-mirror.test.log 2023-11-24T08:42:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ date +%F %T 2023-11-24T08:42:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2023-11-24 08:42:11 TEST: create many images 2023-11-24T08:42:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2023-11-24 08:42:11 TEST: create many images 2023-11-24T08:42:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ snap_name=snap 2023-11-24T08:42:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 50 2023-11-24T08:42:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2023-11-24T08:42:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-11-24T08:42:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_1 128M 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:42:11.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:42:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:42:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:42:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:11.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-11-24T08:42:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:42:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-11-24T08:42:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_1': 2023-11-24T08:42:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:42:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:42:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:42:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 125091295c3b 2023-11-24T08:42:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.125091295c3b 2023-11-24T08:42:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:11 2023 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:11 2023 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:11 2023 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 31318b39-6acb-4741-8297-a101c3fafabc 2023-11-24T08:42:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:42:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:42:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_1 2023-11-24T08:42:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:42:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:42:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2023-11-24T08:42:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:42:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:42:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 32632832 bytes 2023-11-24T08:42:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 104132608 bytes 2023-11-24T08:42:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 168980480 bytes 2023-11-24T08:42:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 192560128 bytes 2023-11-24T08:42:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 257742848 bytes 2023-11-24T08:42:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 275976192 bytes 2023-11-24T08:42:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 305522688 bytes 2023-11-24T08:42:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 339117056 bytes 2023-11-24T08:42:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 361988096 bytes 2023-11-24T08:42:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 396021760 bytes 2023-11-24T08:42:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 445561856 bytes 2023-11-24T08:42:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 448980992 bytes 2023-11-24T08:42:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 506800128 bytes 2023-11-24T08:42:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 559869952 bytes 2023-11-24T08:42:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 619823104 bytes 2023-11-24T08:42:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 639948800 bytes 2023-11-24T08:42:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 706421760 bytes 2023-11-24T08:42:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 707070976 bytes 2023-11-24T08:42:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 717338624 bytes 2023-11-24T08:42:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 733067264 bytes 2023-11-24T08:42:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 796609536 bytes 2023-11-24T08:42:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 812215296 bytes 2023-11-24T08:42:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 861019136 bytes 2023-11-24T08:42:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 872897536 bytes 2023-11-24T08:42:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 884571136 bytes 2023-11-24T08:42:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 913444864 bytes 2023-11-24T08:42:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 917115904 bytes 2023-11-24T08:42:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 948326400 bytes 2023-11-24T08:42:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 993544192 bytes 2023-11-24T08:42:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1041700864 bytes 2023-11-24T08:42:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1079931904 bytes 2023-11-24T08:42:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1107418112 bytes 2023-11-24T08:42:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1129522176 bytes 2023-11-24T08:42:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1157990400 bytes 2023-11-24T08:42:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1181426688 bytes 2023-11-24T08:42:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1212811264 bytes 2023-11-24T08:42:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1265166336 bytes 2023-11-24T08:42:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1265777664 bytes 2023-11-24T08:42:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1302681600 bytes 2023-11-24T08:42:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1311645696 bytes 2023-11-24T08:42:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1368993792 bytes 2023-11-24T08:42:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1391378432 bytes 2023-11-24T08:42:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1436423168 bytes 2023-11-24T08:42:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1473651712 bytes 2023-11-24T08:42:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1474287616 bytes 2023-11-24T08:42:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1497984000 bytes 2023-11-24T08:42:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1561580544 bytes 2023-11-24T08:42:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1608963072 bytes 2023-11-24T08:42:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1663848448 bytes 2023-11-24T08:42:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1664467968 bytes 2023-11-24T08:42:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1711671296 bytes 2023-11-24T08:42:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1735929856 bytes 2023-11-24T08:42:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1810068480 bytes 2023-11-24T08:42:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 1865005056 bytes 2023-11-24T08:42:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 1940853760 bytes 2023-11-24T08:42:20.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 1973302272 bytes 2023-11-24T08:42:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2015546368 bytes 2023-11-24T08:42:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2038166528 bytes 2023-11-24T08:42:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2103138304 bytes 2023-11-24T08:42:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2174344192 bytes 2023-11-24T08:42:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2199109632 bytes 2023-11-24T08:42:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2238806016 bytes 2023-11-24T08:42:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2269076480 bytes 2023-11-24T08:42:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2304508928 bytes 2023-11-24T08:42:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2356518912 bytes 2023-11-24T08:42:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2429092864 bytes 2023-11-24T08:42:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2468362240 bytes 2023-11-24T08:42:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2511744000 bytes 2023-11-24T08:42:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2520361984 bytes 2023-11-24T08:42:23.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2619959296 bytes 2023-11-24T08:42:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2717456384 bytes 2023-11-24T08:42:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2725028864 bytes 2023-11-24T08:42:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2734818304 bytes 2023-11-24T08:42:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2808298496 bytes 2023-11-24T08:42:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2867658752 bytes 2023-11-24T08:42:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 2934563840 bytes 2023-11-24T08:42:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 2989263872 bytes 2023-11-24T08:42:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3040024576 bytes 2023-11-24T08:42:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3108424704 bytes 2023-11-24T08:42:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3140087808 bytes 2023-11-24T08:42:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3185217536 bytes 2023-11-24T08:42:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3239166976 bytes 2023-11-24T08:42:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8300 IOs, 3279910912 bytes 2023-11-24T08:42:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8400 IOs, 3316214784 bytes 2023-11-24T08:42:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:42:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_2 128M 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-11-24T08:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-11-24T08:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:42:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-11-24T08:42:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:42:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-11-24T08:42:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_2': 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 125cadb97733 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.125cadb97733 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:26 2023 2023-11-24T08:42:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:26 2023 2023-11-24T08:42:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:26 2023 2023-11-24T08:42:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:42:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:42:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f38cb77f-9d83-40c5-a0c0-ff19cc94a15c 2023-11-24T08:42:26.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_2 2023-11-24T08:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:42:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:42:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T08:42:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:42:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:42:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 31958016 bytes 2023-11-24T08:42:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 108807168 bytes 2023-11-24T08:42:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 138680320 bytes 2023-11-24T08:42:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 162881536 bytes 2023-11-24T08:42:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 192004096 bytes 2023-11-24T08:42:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 239341568 bytes 2023-11-24T08:42:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 283814912 bytes 2023-11-24T08:42:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 332006400 bytes 2023-11-24T08:42:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 346445824 bytes 2023-11-24T08:42:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 386004992 bytes 2023-11-24T08:42:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 388976640 bytes 2023-11-24T08:42:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 440262656 bytes 2023-11-24T08:42:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 469985280 bytes 2023-11-24T08:42:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 492353536 bytes 2023-11-24T08:42:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 520380416 bytes 2023-11-24T08:42:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 520984576 bytes 2023-11-24T08:42:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 575941632 bytes 2023-11-24T08:42:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 681364480 bytes 2023-11-24T08:42:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 754309120 bytes 2023-11-24T08:42:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 791649280 bytes 2023-11-24T08:42:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 821129216 bytes 2023-11-24T08:42:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 837176320 bytes 2023-11-24T08:42:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 882423808 bytes 2023-11-24T08:42:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 893640704 bytes 2023-11-24T08:42:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 895614976 bytes 2023-11-24T08:42:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 912196608 bytes 2023-11-24T08:42:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 945709056 bytes 2023-11-24T08:42:31.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 953352192 bytes 2023-11-24T08:42:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 996345856 bytes 2023-11-24T08:42:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1036848128 bytes 2023-11-24T08:42:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1076177920 bytes 2023-11-24T08:42:31.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1093184512 bytes 2023-11-24T08:42:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1093799936 bytes 2023-11-24T08:42:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1173204992 bytes 2023-11-24T08:42:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1250549760 bytes 2023-11-24T08:42:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1308676096 bytes 2023-11-24T08:42:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1331476480 bytes 2023-11-24T08:42:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1435373568 bytes 2023-11-24T08:42:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1458031616 bytes 2023-11-24T08:42:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1479655424 bytes 2023-11-24T08:42:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1521647616 bytes 2023-11-24T08:42:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1552754688 bytes 2023-11-24T08:42:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1589136384 bytes 2023-11-24T08:42:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1713280000 bytes 2023-11-24T08:42:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1713867776 bytes 2023-11-24T08:42:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1757043712 bytes 2023-11-24T08:42:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1775407104 bytes 2023-11-24T08:42:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1813519360 bytes 2023-11-24T08:42:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1825523712 bytes 2023-11-24T08:42:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1878689792 bytes 2023-11-24T08:42:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1893672960 bytes 2023-11-24T08:42:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1913390080 bytes 2023-11-24T08:42:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1929042944 bytes 2023-11-24T08:42:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 1993117696 bytes 2023-11-24T08:42:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2043615232 bytes 2023-11-24T08:42:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2102755328 bytes 2023-11-24T08:42:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2117202944 bytes 2023-11-24T08:42:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2159516672 bytes 2023-11-24T08:42:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2189931520 bytes 2023-11-24T08:42:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2212685824 bytes 2023-11-24T08:42:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2213285888 bytes 2023-11-24T08:42:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2223547392 bytes 2023-11-24T08:42:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:42:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:42:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:42:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2023-11-24T08:42:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-11-24T08:42:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:42:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:42:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_3 128M 2023-11-24T08:42:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:42:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:42:36.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:42:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:42:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-11-24T08:42:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:42:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:42:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-11-24T08:42:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:42:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-11-24T08:42:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:42:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-11-24T08:42:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_3': 2023-11-24T08:42:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:42:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:42:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:42:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 126847088c0f 2023-11-24T08:42:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.126847088c0f 2023-11-24T08:42:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:42:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:42:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:42:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:42:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:36 2023 2023-11-24T08:42:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:36 2023 2023-11-24T08:42:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:36 2023 2023-11-24T08:42:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:42:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:42:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 0910fa58-c0e2-4ebc-ba44-f7fe28530964 2023-11-24T08:42:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:42:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:42:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_3 2023-11-24T08:42:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:42:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:42:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2023-11-24T08:42:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:42:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:42:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 35051520 bytes 2023-11-24T08:42:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 72826880 bytes 2023-11-24T08:42:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 145973248 bytes 2023-11-24T08:42:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 166077440 bytes 2023-11-24T08:42:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 269145088 bytes 2023-11-24T08:42:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 322345984 bytes 2023-11-24T08:42:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 351322112 bytes 2023-11-24T08:42:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 372009984 bytes 2023-11-24T08:42:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 395489280 bytes 2023-11-24T08:42:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 411236352 bytes 2023-11-24T08:42:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 444638208 bytes 2023-11-24T08:42:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 449781760 bytes 2023-11-24T08:42:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 500893696 bytes 2023-11-24T08:42:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 529673216 bytes 2023-11-24T08:42:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 561456128 bytes 2023-11-24T08:42:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 580624384 bytes 2023-11-24T08:42:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 597605376 bytes 2023-11-24T08:42:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 647548928 bytes 2023-11-24T08:42:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 674649088 bytes 2023-11-24T08:42:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 707693568 bytes 2023-11-24T08:42:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 779624448 bytes 2023-11-24T08:42:41.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 867412992 bytes 2023-11-24T08:42:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 898150400 bytes 2023-11-24T08:42:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 924973056 bytes 2023-11-24T08:42:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 949391360 bytes 2023-11-24T08:42:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1003833344 bytes 2023-11-24T08:42:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1056214016 bytes 2023-11-24T08:42:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1096307712 bytes 2023-11-24T08:42:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1144833024 bytes 2023-11-24T08:42:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1182644224 bytes 2023-11-24T08:42:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1241237504 bytes 2023-11-24T08:42:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1298113536 bytes 2023-11-24T08:42:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1344971776 bytes 2023-11-24T08:42:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1358731264 bytes 2023-11-24T08:42:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:42:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:42:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:42:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_4 128M 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-11-24T08:42:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:42:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:42:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-11-24T08:42:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:42:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-11-24T08:42:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:42:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-11-24T08:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_4': 2023-11-24T08:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12741fb34402 2023-11-24T08:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12741fb34402 2023-11-24T08:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:42:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:42:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:42:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:42:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:43 2023 2023-11-24T08:42:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:43 2023 2023-11-24T08:42:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:43 2023 2023-11-24T08:42:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:42:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:42:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: eb96c510-4f18-49d6-8bef-2744c8373fbf 2023-11-24T08:42:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_4 2023-11-24T08:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:42:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T08:42:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:42:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:42:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 53850112 bytes 2023-11-24T08:42:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 70902784 bytes 2023-11-24T08:42:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 142443520 bytes 2023-11-24T08:42:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 172346368 bytes 2023-11-24T08:42:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 182759424 bytes 2023-11-24T08:42:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 210672640 bytes 2023-11-24T08:42:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 254422016 bytes 2023-11-24T08:42:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 310003712 bytes 2023-11-24T08:42:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 351366144 bytes 2023-11-24T08:42:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 389196800 bytes 2023-11-24T08:42:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 414935040 bytes 2023-11-24T08:42:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 476492800 bytes 2023-11-24T08:42:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 533686272 bytes 2023-11-24T08:42:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 539524096 bytes 2023-11-24T08:42:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 589612032 bytes 2023-11-24T08:42:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 622733312 bytes 2023-11-24T08:42:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 664010752 bytes 2023-11-24T08:42:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 756219904 bytes 2023-11-24T08:42:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 802207744 bytes 2023-11-24T08:42:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 885353472 bytes 2023-11-24T08:42:48.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 949638144 bytes 2023-11-24T08:42:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 971029504 bytes 2023-11-24T08:42:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 987146240 bytes 2023-11-24T08:42:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1007073280 bytes 2023-11-24T08:42:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1028147200 bytes 2023-11-24T08:42:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1095231488 bytes 2023-11-24T08:42:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1155537920 bytes 2023-11-24T08:42:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1218861056 bytes 2023-11-24T08:42:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1259488256 bytes 2023-11-24T08:42:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1354043392 bytes 2023-11-24T08:42:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1380058112 bytes 2023-11-24T08:42:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1410446336 bytes 2023-11-24T08:42:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1448464384 bytes 2023-11-24T08:42:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1497863168 bytes 2023-11-24T08:42:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1536667648 bytes 2023-11-24T08:42:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1619406848 bytes 2023-11-24T08:42:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1721711616 bytes 2023-11-24T08:42:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:42:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:42:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_5 128M 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:42:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:42:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:42:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:42:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-11-24T08:42:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-11-24T08:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:42:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-11-24T08:42:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:42:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-11-24T08:42:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_5': 2023-11-24T08:42:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:42:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:42:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 128071d50cbc 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.128071d50cbc 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:51 2023 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:51 2023 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:51 2023 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:42:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 14adee4f-4c4d-4e0f-8207-88b9d7b5b7af 2023-11-24T08:42:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_5 2023-11-24T08:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2023-11-24T08:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:42:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:42:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 102875136 bytes 2023-11-24T08:42:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 148265984 bytes 2023-11-24T08:42:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 164173824 bytes 2023-11-24T08:42:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 167465984 bytes 2023-11-24T08:42:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 168713216 bytes 2023-11-24T08:42:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 221942784 bytes 2023-11-24T08:42:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 243312640 bytes 2023-11-24T08:42:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 281245696 bytes 2023-11-24T08:42:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 326603776 bytes 2023-11-24T08:42:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 392370176 bytes 2023-11-24T08:42:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 411535360 bytes 2023-11-24T08:42:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 457625600 bytes 2023-11-24T08:42:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 458231808 bytes 2023-11-24T08:42:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 515334144 bytes 2023-11-24T08:42:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 578350080 bytes 2023-11-24T08:42:55.468 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9011746079805512) > weight (0.25) 2023-11-24T08:42:55.468 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.42298111304760666) > weight (0.25) 2023-11-24T08:42:55.468 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8842906924932269) > weight (0.25) 2023-11-24T08:42:55.468 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-24T08:42:55.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-24T08:42:55.469 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-24T08:42:55.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T08:42:55.467+0000 7f5c6e721640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 46979) UID: 1000 2023-11-24T08:42:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 610225152 bytes 2023-11-24T08:42:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 701786112 bytes 2023-11-24T08:42:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 771825664 bytes 2023-11-24T08:42:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 796424192 bytes 2023-11-24T08:42:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 822324224 bytes 2023-11-24T08:42:56.469 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-24T08:42:56.469 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:42:56.470 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-24T08:42:56.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-24T08:42:56.470 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-24T08:42:56.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-24T08:42:56.473 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2023-11-24T08:42:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 901571584 bytes 2023-11-24T08:42:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 967524352 bytes 2023-11-24T08:42:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1002939392 bytes 2023-11-24T08:42:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1032552448 bytes 2023-11-24T08:42:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1072396288 bytes 2023-11-24T08:42:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1104446464 bytes 2023-11-24T08:42:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1140540416 bytes 2023-11-24T08:42:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1228221440 bytes 2023-11-24T08:42:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1260205056 bytes 2023-11-24T08:42:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1287024640 bytes 2023-11-24T08:42:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1331895296 bytes 2023-11-24T08:42:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1351754752 bytes 2023-11-24T08:42:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1429272576 bytes 2023-11-24T08:42:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1457404928 bytes 2023-11-24T08:42:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1506156544 bytes 2023-11-24T08:42:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1572145152 bytes 2023-11-24T08:42:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1616603136 bytes 2023-11-24T08:42:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1684005888 bytes 2023-11-24T08:42:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1751013376 bytes 2023-11-24T08:43:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1807657984 bytes 2023-11-24T08:43:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1892021248 bytes 2023-11-24T08:43:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1955113984 bytes 2023-11-24T08:43:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1986760704 bytes 2023-11-24T08:43:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1992775680 bytes 2023-11-24T08:43:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2031254528 bytes 2023-11-24T08:43:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2051385344 bytes 2023-11-24T08:43:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2099211264 bytes 2023-11-24T08:43:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2121717760 bytes 2023-11-24T08:43:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2153263104 bytes 2023-11-24T08:43:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2257844224 bytes 2023-11-24T08:43:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2323439616 bytes 2023-11-24T08:43:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2347335680 bytes 2023-11-24T08:43:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2359093248 bytes 2023-11-24T08:43:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2390121472 bytes 2023-11-24T08:43:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2420971520 bytes 2023-11-24T08:43:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2452923392 bytes 2023-11-24T08:43:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2458516480 bytes 2023-11-24T08:43:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2509230080 bytes 2023-11-24T08:43:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2525687808 bytes 2023-11-24T08:43:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2546705408 bytes 2023-11-24T08:43:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2596234240 bytes 2023-11-24T08:43:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2618424320 bytes 2023-11-24T08:43:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2638282752 bytes 2023-11-24T08:43:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2663165952 bytes 2023-11-24T08:43:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2710246400 bytes 2023-11-24T08:43:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2731067392 bytes 2023-11-24T08:43:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2818390016 bytes 2023-11-24T08:43:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2831651840 bytes 2023-11-24T08:43:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2890710016 bytes 2023-11-24T08:43:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2954498048 bytes 2023-11-24T08:43:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2974687232 bytes 2023-11-24T08:43:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2981684224 bytes 2023-11-24T08:43:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3092346880 bytes 2023-11-24T08:43:05.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3129112576 bytes 2023-11-24T08:43:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3140070400 bytes 2023-11-24T08:43:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3164508160 bytes 2023-11-24T08:43:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7700 IOs, 3172319232 bytes 2023-11-24T08:43:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7800 IOs, 3190559744 bytes 2023-11-24T08:43:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7900 IOs, 3194884096 bytes 2023-11-24T08:43:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8000 IOs, 3225498624 bytes 2023-11-24T08:43:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8100 IOs, 3249348608 bytes 2023-11-24T08:43:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:8200 IOs, 3284164608 bytes 2023-11-24T08:43:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:43:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:43:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:43:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_6 128M 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-11-24T08:43:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:43:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:43:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-11-24T08:43:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:43:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-11-24T08:43:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:43:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-11-24T08:43:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_6': 2023-11-24T08:43:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:43:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:43:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:43:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1296ab522269 2023-11-24T08:43:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1296ab522269 2023-11-24T08:43:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:43:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:43:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:43:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:43:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:06 2023 2023-11-24T08:43:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:06 2023 2023-11-24T08:43:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:06 2023 2023-11-24T08:43:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:43:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:43:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f792a31d-dcf2-4828-bc5f-9490ec637a03 2023-11-24T08:43:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:43:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:43:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_6 2023-11-24T08:43:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:43:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:43:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2023-11-24T08:43:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:43:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:43:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 31081472 bytes 2023-11-24T08:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 43808768 bytes 2023-11-24T08:43:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 119555072 bytes 2023-11-24T08:43:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 203574272 bytes 2023-11-24T08:43:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 281807872 bytes 2023-11-24T08:43:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 299708416 bytes 2023-11-24T08:43:08.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 339312640 bytes 2023-11-24T08:43:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 354179072 bytes 2023-11-24T08:43:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 456017920 bytes 2023-11-24T08:43:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 604469248 bytes 2023-11-24T08:43:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 622229504 bytes 2023-11-24T08:43:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 638675968 bytes 2023-11-24T08:43:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 715791360 bytes 2023-11-24T08:43:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 734875648 bytes 2023-11-24T08:43:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 816272384 bytes 2023-11-24T08:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 871464960 bytes 2023-11-24T08:43:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 881710080 bytes 2023-11-24T08:43:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 950270976 bytes 2023-11-24T08:43:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1018061824 bytes 2023-11-24T08:43:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1059400704 bytes 2023-11-24T08:43:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1108732928 bytes 2023-11-24T08:43:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1119306752 bytes 2023-11-24T08:43:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1167697920 bytes 2023-11-24T08:43:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1190011904 bytes 2023-11-24T08:43:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1200422912 bytes 2023-11-24T08:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1245901824 bytes 2023-11-24T08:43:12.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1275673600 bytes 2023-11-24T08:43:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1315138560 bytes 2023-11-24T08:43:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1367024640 bytes 2023-11-24T08:43:13.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1404197888 bytes 2023-11-24T08:43:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1472070656 bytes 2023-11-24T08:43:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1473426432 bytes 2023-11-24T08:43:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1599149056 bytes 2023-11-24T08:43:13.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1599766528 bytes 2023-11-24T08:43:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1638078464 bytes 2023-11-24T08:43:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1713316864 bytes 2023-11-24T08:43:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1768903680 bytes 2023-11-24T08:43:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1792346112 bytes 2023-11-24T08:43:14.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1822585856 bytes 2023-11-24T08:43:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1894569984 bytes 2023-11-24T08:43:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1898983424 bytes 2023-11-24T08:43:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1942814720 bytes 2023-11-24T08:43:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1976713216 bytes 2023-11-24T08:43:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2021071872 bytes 2023-11-24T08:43:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2048415744 bytes 2023-11-24T08:43:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2080091136 bytes 2023-11-24T08:43:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2127967232 bytes 2023-11-24T08:43:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:43:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:43:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:43:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2023-11-24T08:43:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_7 128M 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:43:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:43:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:43:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:43:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-11-24T08:43:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:43:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:43:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-11-24T08:43:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:43:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-11-24T08:43:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:43:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_7': 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12a2e3e085f1 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12a2e3e085f1 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:43:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:16 2023 2023-11-24T08:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:16 2023 2023-11-24T08:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:16 2023 2023-11-24T08:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: e07e8277-39ef-4a01-8ed4-6a47771afe3f 2023-11-24T08:43:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_7 2023-11-24T08:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:43:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2023-11-24T08:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:43:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:43:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 48833536 bytes 2023-11-24T08:43:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 78235648 bytes 2023-11-24T08:43:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 110848000 bytes 2023-11-24T08:43:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 196564992 bytes 2023-11-24T08:43:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 238097408 bytes 2023-11-24T08:43:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 285584384 bytes 2023-11-24T08:43:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 320412672 bytes 2023-11-24T08:43:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 373454848 bytes 2023-11-24T08:43:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 406053888 bytes 2023-11-24T08:43:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 529686528 bytes 2023-11-24T08:43:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 594339840 bytes 2023-11-24T08:43:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 626964480 bytes 2023-11-24T08:43:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 722877440 bytes 2023-11-24T08:43:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 760386560 bytes 2023-11-24T08:43:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 878276608 bytes 2023-11-24T08:43:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 913463296 bytes 2023-11-24T08:43:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 973663232 bytes 2023-11-24T08:43:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 1030256640 bytes 2023-11-24T08:43:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1117520896 bytes 2023-11-24T08:43:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1167110144 bytes 2023-11-24T08:43:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1203420160 bytes 2023-11-24T08:43:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1264818176 bytes 2023-11-24T08:43:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1327484928 bytes 2023-11-24T08:43:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1328103424 bytes 2023-11-24T08:43:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1370655744 bytes 2023-11-24T08:43:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1394851840 bytes 2023-11-24T08:43:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1407722496 bytes 2023-11-24T08:43:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1430465536 bytes 2023-11-24T08:43:22.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1510491136 bytes 2023-11-24T08:43:22.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1550039040 bytes 2023-11-24T08:43:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1617341440 bytes 2023-11-24T08:43:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1681483776 bytes 2023-11-24T08:43:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1737851904 bytes 2023-11-24T08:43:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1760156672 bytes 2023-11-24T08:43:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1785489408 bytes 2023-11-24T08:43:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1811320832 bytes 2023-11-24T08:43:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1825788928 bytes 2023-11-24T08:43:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1889924096 bytes 2023-11-24T08:43:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1971701760 bytes 2023-11-24T08:43:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1996745728 bytes 2023-11-24T08:43:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 2045553664 bytes 2023-11-24T08:43:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2068548608 bytes 2023-11-24T08:43:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2156632064 bytes 2023-11-24T08:43:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2199902208 bytes 2023-11-24T08:43:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2300582912 bytes 2023-11-24T08:43:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2379691008 bytes 2023-11-24T08:43:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2422644736 bytes 2023-11-24T08:43:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2521290752 bytes 2023-11-24T08:43:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2595252224 bytes 2023-11-24T08:43:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2641308672 bytes 2023-11-24T08:43:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2711984128 bytes 2023-11-24T08:43:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2737033216 bytes 2023-11-24T08:43:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2738316288 bytes 2023-11-24T08:43:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2799193088 bytes 2023-11-24T08:43:27.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2844756992 bytes 2023-11-24T08:43:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2855840768 bytes 2023-11-24T08:43:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2932600832 bytes 2023-11-24T08:43:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2968679424 bytes 2023-11-24T08:43:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2991148032 bytes 2023-11-24T08:43:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 3022486528 bytes 2023-11-24T08:43:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 3029577728 bytes 2023-11-24T08:43:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 3061348352 bytes 2023-11-24T08:43:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 3107431424 bytes 2023-11-24T08:43:29.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 3174073344 bytes 2023-11-24T08:43:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 3220090880 bytes 2023-11-24T08:43:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3263239168 bytes 2023-11-24T08:43:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3392769024 bytes 2023-11-24T08:43:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:43:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:43:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:43:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2023-11-24T08:43:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-11-24T08:43:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:43:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:43:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_8 128M 2023-11-24T08:43:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:43:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:43:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:43:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-11-24T08:43:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:43:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:43:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-11-24T08:43:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:43:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-11-24T08:43:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:43:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-11-24T08:43:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_8': 2023-11-24T08:43:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:43:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:43:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:43:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12aea0fb54ec 2023-11-24T08:43:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12aea0fb54ec 2023-11-24T08:43:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:43:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:43:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:43:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:43:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:29 2023 2023-11-24T08:43:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:29 2023 2023-11-24T08:43:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:29 2023 2023-11-24T08:43:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:43:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:43:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 8b099305-4776-4fa1-8f3c-5e18bc48f849 2023-11-24T08:43:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:43:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:43:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_8 2023-11-24T08:43:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:43:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:43:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2023-11-24T08:43:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:43:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:43:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 8802304 bytes 2023-11-24T08:43:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 83868672 bytes 2023-11-24T08:43:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 149149696 bytes 2023-11-24T08:43:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 192616448 bytes 2023-11-24T08:43:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 227369984 bytes 2023-11-24T08:43:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 268348416 bytes 2023-11-24T08:43:32.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 268953600 bytes 2023-11-24T08:43:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 318372864 bytes 2023-11-24T08:43:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 382978048 bytes 2023-11-24T08:43:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 387840000 bytes 2023-11-24T08:43:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 461352960 bytes 2023-11-24T08:43:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 543686656 bytes 2023-11-24T08:43:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 650225664 bytes 2023-11-24T08:43:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 711035904 bytes 2023-11-24T08:43:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 736418816 bytes 2023-11-24T08:43:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 814576640 bytes 2023-11-24T08:43:34.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 931607552 bytes 2023-11-24T08:43:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 1017702400 bytes 2023-11-24T08:43:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1031783424 bytes 2023-11-24T08:43:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1124838400 bytes 2023-11-24T08:43:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1148424192 bytes 2023-11-24T08:43:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1183601664 bytes 2023-11-24T08:43:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1261298688 bytes 2023-11-24T08:43:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1298604032 bytes 2023-11-24T08:43:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1352074240 bytes 2023-11-24T08:43:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1378145280 bytes 2023-11-24T08:43:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1454058496 bytes 2023-11-24T08:43:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1495798784 bytes 2023-11-24T08:43:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:43:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:43:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:43:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_9 128M 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-11-24T08:43:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:43:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:43:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-11-24T08:43:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:43:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-11-24T08:43:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:43:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-11-24T08:43:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_9': 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12babf17aba5 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12babf17aba5 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:43:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:36 2023 2023-11-24T08:43:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:36 2023 2023-11-24T08:43:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:36 2023 2023-11-24T08:43:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:43:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:43:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 18de629c-76f5-4bac-9f50-e3a0a137e23c 2023-11-24T08:43:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:43:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:43:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_9 2023-11-24T08:43:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:43:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:43:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2023-11-24T08:43:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:43:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:43:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 40996864 bytes 2023-11-24T08:43:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 82356224 bytes 2023-11-24T08:43:38.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 83060736 bytes 2023-11-24T08:43:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 132673536 bytes 2023-11-24T08:43:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 196830208 bytes 2023-11-24T08:43:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 259318784 bytes 2023-11-24T08:43:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 290232320 bytes 2023-11-24T08:43:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 316380160 bytes 2023-11-24T08:43:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 340618240 bytes 2023-11-24T08:43:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 363165696 bytes 2023-11-24T08:43:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 379758592 bytes 2023-11-24T08:43:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 489043968 bytes 2023-11-24T08:43:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 491953152 bytes 2023-11-24T08:43:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 562833408 bytes 2023-11-24T08:43:40.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 592771072 bytes 2023-11-24T08:43:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 636358656 bytes 2023-11-24T08:43:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 695770112 bytes 2023-11-24T08:43:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 824928256 bytes 2023-11-24T08:43:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 875594752 bytes 2023-11-24T08:43:41.582 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5954795223257414) > weight (0.25) 2023-11-24T08:43:41.582 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.2554142014152905) > weight (0.25) 2023-11-24T08:43:41.582 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8449372465242988) > weight (0.25) 2023-11-24T08:43:41.582 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3553470748275559) > weight (0.25) 2023-11-24T08:43:41.582 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-11-24T08:43:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 940005376 bytes 2023-11-24T08:43:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 959327232 bytes 2023-11-24T08:43:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1040233472 bytes 2023-11-24T08:43:42.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1117389824 bytes 2023-11-24T08:43:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1175862272 bytes 2023-11-24T08:43:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1219671040 bytes 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:42.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_10 128M 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-11-24T08:43:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:43:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-11-24T08:43:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_10': 2023-11-24T08:43:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12c694cf1514 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12c694cf1514 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:42 2023 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:42 2023 2023-11-24T08:43:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:42 2023 2023-11-24T08:43:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:43:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:43:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 342ef9c2-57fd-4085-be72-fae4ddc7c8a0 2023-11-24T08:43:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:43:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:43:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_10 2023-11-24T08:43:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:43:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2023-11-24T08:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:43:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:43:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 32755712 bytes 2023-11-24T08:43:44.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 50930688 bytes 2023-11-24T08:43:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 113865728 bytes 2023-11-24T08:43:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 167659520 bytes 2023-11-24T08:43:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 214176768 bytes 2023-11-24T08:43:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 241020928 bytes 2023-11-24T08:43:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 277173248 bytes 2023-11-24T08:43:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 343498752 bytes 2023-11-24T08:43:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 364734464 bytes 2023-11-24T08:43:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 459121664 bytes 2023-11-24T08:43:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 510777344 bytes 2023-11-24T08:43:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 586046464 bytes 2023-11-24T08:43:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 593943552 bytes 2023-11-24T08:43:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 622398464 bytes 2023-11-24T08:43:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 650477568 bytes 2023-11-24T08:43:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 651091968 bytes 2023-11-24T08:43:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 692487168 bytes 2023-11-24T08:43:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 725989376 bytes 2023-11-24T08:43:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 779948032 bytes 2023-11-24T08:43:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 780575744 bytes 2023-11-24T08:43:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 790777856 bytes 2023-11-24T08:43:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 854950912 bytes 2023-11-24T08:43:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 912932864 bytes 2023-11-24T08:43:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 938680320 bytes 2023-11-24T08:43:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 993629184 bytes 2023-11-24T08:43:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1002139648 bytes 2023-11-24T08:43:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1033714688 bytes 2023-11-24T08:43:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1095240704 bytes 2023-11-24T08:43:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1179681792 bytes 2023-11-24T08:43:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1245424640 bytes 2023-11-24T08:43:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1298963456 bytes 2023-11-24T08:43:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1315858432 bytes 2023-11-24T08:43:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1316486144 bytes 2023-11-24T08:43:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1370809344 bytes 2023-11-24T08:43:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1419596800 bytes 2023-11-24T08:43:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1530708992 bytes 2023-11-24T08:43:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1583996928 bytes 2023-11-24T08:43:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1607544832 bytes 2023-11-24T08:43:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1644595200 bytes 2023-11-24T08:43:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1686023168 bytes 2023-11-24T08:43:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1746646016 bytes 2023-11-24T08:43:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1796236288 bytes 2023-11-24T08:43:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1799077888 bytes 2023-11-24T08:43:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1856845824 bytes 2023-11-24T08:43:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1857460224 bytes 2023-11-24T08:43:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1893954560 bytes 2023-11-24T08:43:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1915922432 bytes 2023-11-24T08:43:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1958074368 bytes 2023-11-24T08:43:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1978681344 bytes 2023-11-24T08:43:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:43:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_11 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:52.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_11 128M 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:43:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2023-11-24T08:43:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:43:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:43:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:43:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-11-24T08:43:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:43:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:43:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-11-24T08:43:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:43:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-11-24T08:43:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_11': 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12d14e2674c3 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12d14e2674c3 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:43:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:52 2023 2023-11-24T08:43:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:52 2023 2023-11-24T08:43:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:52 2023 2023-11-24T08:43:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:43:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:43:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 93be2a75-cd83-40a7-a3da-37d47064a00c 2023-11-24T08:43:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:43:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:43:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_11 2023-11-24T08:43:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:43:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:43:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:43:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:43:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2023-11-24T08:43:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:43:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:43:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 30976000 bytes 2023-11-24T08:43:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 80857088 bytes 2023-11-24T08:43:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 133332992 bytes 2023-11-24T08:43:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 187096064 bytes 2023-11-24T08:43:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 191572992 bytes 2023-11-24T08:43:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 271386624 bytes 2023-11-24T08:43:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 328109056 bytes 2023-11-24T08:43:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 370905088 bytes 2023-11-24T08:43:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 417310720 bytes 2023-11-24T08:43:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 476909568 bytes 2023-11-24T08:43:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 493938688 bytes 2023-11-24T08:43:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 542540800 bytes 2023-11-24T08:43:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 637989888 bytes 2023-11-24T08:43:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 677128192 bytes 2023-11-24T08:43:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 677751808 bytes 2023-11-24T08:43:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 772617216 bytes 2023-11-24T08:43:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 813745152 bytes 2023-11-24T08:43:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 818241536 bytes 2023-11-24T08:43:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 830838784 bytes 2023-11-24T08:43:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 907528192 bytes 2023-11-24T08:43:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 969400320 bytes 2023-11-24T08:43:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1046629376 bytes 2023-11-24T08:43:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1073992704 bytes 2023-11-24T08:43:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1118619648 bytes 2023-11-24T08:43:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1124866048 bytes 2023-11-24T08:43:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1127897088 bytes 2023-11-24T08:43:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1186944000 bytes 2023-11-24T08:43:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1250053120 bytes 2023-11-24T08:43:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1278024704 bytes 2023-11-24T08:43:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1325421568 bytes 2023-11-24T08:43:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1371144192 bytes 2023-11-24T08:43:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1443821568 bytes 2023-11-24T08:43:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1450570752 bytes 2023-11-24T08:44:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1515380736 bytes 2023-11-24T08:44:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1516014592 bytes 2023-11-24T08:44:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1612396544 bytes 2023-11-24T08:44:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1663912960 bytes 2023-11-24T08:44:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1722497024 bytes 2023-11-24T08:44:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1737237504 bytes 2023-11-24T08:44:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1789298688 bytes 2023-11-24T08:44:01.474 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5046369238075722) > weight (0.25) 2023-11-24T08:44:01.474 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-24T08:44:01.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-24T08:44:01.475 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-24T08:44:01.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T08:44:01.478+0000 7f4f49741640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 45654) UID: 1000 2023-11-24T08:44:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1840653312 bytes 2023-11-24T08:44:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1850982400 bytes 2023-11-24T08:44:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1871820800 bytes 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_12 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_12 128M 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:01.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:44:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:44:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:44:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2023-11-24T08:44:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:44:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:44:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-11-24T08:44:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:44:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:01.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-11-24T08:44:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:44:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-11-24T08:44:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_12': 2023-11-24T08:44:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:44:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:44:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:44:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12dcb7fdd426 2023-11-24T08:44:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12dcb7fdd426 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:01 2023 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:01 2023 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:01 2023 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 08429c44-38df-4fe3-933e-3a53c6f058ba 2023-11-24T08:44:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:44:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:44:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_12 2023-11-24T08:44:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:02.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:44:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:44:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2023-11-24T08:44:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:44:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:44:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 35661824 bytes 2023-11-24T08:44:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 47810560 bytes 2023-11-24T08:44:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 104877056 bytes 2023-11-24T08:44:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 118560768 bytes 2023-11-24T08:44:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 176120832 bytes 2023-11-24T08:44:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 236183552 bytes 2023-11-24T08:44:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 259555328 bytes 2023-11-24T08:44:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 322245632 bytes 2023-11-24T08:44:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 333385728 bytes 2023-11-24T08:44:04.476 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-24T08:44:04.476 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:44:04.477 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-24T08:44:04.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-24T08:44:04.477 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-24T08:44:04.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-24T08:44:04.479 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2023-11-24T08:44:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 358008832 bytes 2023-11-24T08:44:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 373318656 bytes 2023-11-24T08:44:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 409860096 bytes 2023-11-24T08:44:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 451469312 bytes 2023-11-24T08:44:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 472821760 bytes 2023-11-24T08:44:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 559380480 bytes 2023-11-24T08:44:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 621278208 bytes 2023-11-24T08:44:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 642710528 bytes 2023-11-24T08:44:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 749215744 bytes 2023-11-24T08:44:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 808663040 bytes 2023-11-24T08:44:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 911696896 bytes 2023-11-24T08:44:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 963740672 bytes 2023-11-24T08:44:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1007005696 bytes 2023-11-24T08:44:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1104480256 bytes 2023-11-24T08:44:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1146320896 bytes 2023-11-24T08:44:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1169152000 bytes 2023-11-24T08:44:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1185318912 bytes 2023-11-24T08:44:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:44:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:44:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:44:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_13 2023-11-24T08:44:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-11-24T08:44:07.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_13 128M 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:44:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:44:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:44:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:44:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2023-11-24T08:44:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:44:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:44:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-11-24T08:44:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:44:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-11-24T08:44:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:44:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-11-24T08:44:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_13': 2023-11-24T08:44:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12ece95c02f5 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12ece95c02f5 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:07 2023 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:07 2023 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:07 2023 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:44:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 25e23892-6380-4923-94d3-6e58c6e982f0 2023-11-24T08:44:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:44:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:44:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_13 2023-11-24T08:44:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:44:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:44:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2023-11-24T08:44:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:44:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:44:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 14861312 bytes 2023-11-24T08:44:09.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 84347904 bytes 2023-11-24T08:44:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 84954112 bytes 2023-11-24T08:44:09.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 110183424 bytes 2023-11-24T08:44:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 174193664 bytes 2023-11-24T08:44:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 186966016 bytes 2023-11-24T08:44:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 247558144 bytes 2023-11-24T08:44:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 361497600 bytes 2023-11-24T08:44:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 471278592 bytes 2023-11-24T08:44:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 580528128 bytes 2023-11-24T08:44:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 612676608 bytes 2023-11-24T08:44:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 642867200 bytes 2023-11-24T08:44:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 671777792 bytes 2023-11-24T08:44:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 738649088 bytes 2023-11-24T08:44:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 870174720 bytes 2023-11-24T08:44:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 891091968 bytes 2023-11-24T08:44:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 930484224 bytes 2023-11-24T08:44:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 987292672 bytes 2023-11-24T08:44:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1029627904 bytes 2023-11-24T08:44:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1062005760 bytes 2023-11-24T08:44:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1109411840 bytes 2023-11-24T08:44:13.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1137513472 bytes 2023-11-24T08:44:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1218466816 bytes 2023-11-24T08:44:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1299559424 bytes 2023-11-24T08:44:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1328405504 bytes 2023-11-24T08:44:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1373374464 bytes 2023-11-24T08:44:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1381239808 bytes 2023-11-24T08:44:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1456557056 bytes 2023-11-24T08:44:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1527641088 bytes 2023-11-24T08:44:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1573924864 bytes 2023-11-24T08:44:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1585991680 bytes 2023-11-24T08:44:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1631636480 bytes 2023-11-24T08:44:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1666588672 bytes 2023-11-24T08:44:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1700914176 bytes 2023-11-24T08:44:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1749538816 bytes 2023-11-24T08:44:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1815419904 bytes 2023-11-24T08:44:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1876319232 bytes 2023-11-24T08:44:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1911207936 bytes 2023-11-24T08:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:44:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:44:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_14 2023-11-24T08:44:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-11-24T08:44:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:44:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:44:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_14 128M 2023-11-24T08:44:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:44:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:44:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:44:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:44:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2023-11-24T08:44:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-11-24T08:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-11-24T08:44:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:44:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-11-24T08:44:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_14': 2023-11-24T08:44:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:44:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:44:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:44:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12f62c27ae39 2023-11-24T08:44:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12f62c27ae39 2023-11-24T08:44:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:44:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:44:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:44:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:44:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:16 2023 2023-11-24T08:44:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:16 2023 2023-11-24T08:44:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:16 2023 2023-11-24T08:44:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:44:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:44:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 2cfb09ab-c214-4e47-89a7-d051e9682117 2023-11-24T08:44:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:44:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:44:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_14 2023-11-24T08:44:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:44:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:44:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2023-11-24T08:44:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:44:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:44:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 48011264 bytes 2023-11-24T08:44:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 84157440 bytes 2023-11-24T08:44:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 103257088 bytes 2023-11-24T08:44:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 120846336 bytes 2023-11-24T08:44:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 179777536 bytes 2023-11-24T08:44:19.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 209654784 bytes 2023-11-24T08:44:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 239268864 bytes 2023-11-24T08:44:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 328034304 bytes 2023-11-24T08:44:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 383059968 bytes 2023-11-24T08:44:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 412868608 bytes 2023-11-24T08:44:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 413900800 bytes 2023-11-24T08:44:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 439620608 bytes 2023-11-24T08:44:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 523814912 bytes 2023-11-24T08:44:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 557763584 bytes 2023-11-24T08:44:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 613537792 bytes 2023-11-24T08:44:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 662636544 bytes 2023-11-24T08:44:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 683298816 bytes 2023-11-24T08:44:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 705406976 bytes 2023-11-24T08:44:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 714319872 bytes 2023-11-24T08:44:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 719369216 bytes 2023-11-24T08:44:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 743916544 bytes 2023-11-24T08:44:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 805643264 bytes 2023-11-24T08:44:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 826005504 bytes 2023-11-24T08:44:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 843338752 bytes 2023-11-24T08:44:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 901810176 bytes 2023-11-24T08:44:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 925862912 bytes 2023-11-24T08:44:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 996497408 bytes 2023-11-24T08:44:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1081533440 bytes 2023-11-24T08:44:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:44:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:44:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:44:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_15 2023-11-24T08:44:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-11-24T08:44:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:44:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:44:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_15 128M 2023-11-24T08:44:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:44:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:44:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:44:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:44:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2023-11-24T08:44:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:44:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:44:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-11-24T08:44:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:44:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-11-24T08:44:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:44:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-11-24T08:44:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_15': 2023-11-24T08:44:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:44:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:44:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:44:23.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12ffe59922a 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12ffe59922a 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:22 2023 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:22 2023 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:22 2023 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f1cebaf0-aa03-4dea-9666-f49bdf41a7b2 2023-11-24T08:44:23.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:44:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:44:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_15 2023-11-24T08:44:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:44:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:44:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T08:44:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:44:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:44:24.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 67185664 bytes 2023-11-24T08:44:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 103212032 bytes 2023-11-24T08:44:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 145592320 bytes 2023-11-24T08:44:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 152554496 bytes 2023-11-24T08:44:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 153569280 bytes 2023-11-24T08:44:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 243943424 bytes 2023-11-24T08:44:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 264728576 bytes 2023-11-24T08:44:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 324996096 bytes 2023-11-24T08:44:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 422926336 bytes 2023-11-24T08:44:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 463647744 bytes 2023-11-24T08:44:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 539222016 bytes 2023-11-24T08:44:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 553311232 bytes 2023-11-24T08:44:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 669879296 bytes 2023-11-24T08:44:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 693372928 bytes 2023-11-24T08:44:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 722017280 bytes 2023-11-24T08:44:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 784235520 bytes 2023-11-24T08:44:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 826050560 bytes 2023-11-24T08:44:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 872062976 bytes 2023-11-24T08:44:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 933188608 bytes 2023-11-24T08:44:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 988499968 bytes 2023-11-24T08:44:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1052454912 bytes 2023-11-24T08:44:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1070901248 bytes 2023-11-24T08:44:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1139557376 bytes 2023-11-24T08:44:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1237165056 bytes 2023-11-24T08:44:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1305374720 bytes 2023-11-24T08:44:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1348054016 bytes 2023-11-24T08:44:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1396418560 bytes 2023-11-24T08:44:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1398926336 bytes 2023-11-24T08:44:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1480564736 bytes 2023-11-24T08:44:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1508224000 bytes 2023-11-24T08:44:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1521229824 bytes 2023-11-24T08:44:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1540024320 bytes 2023-11-24T08:44:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1558521856 bytes 2023-11-24T08:44:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1637685248 bytes 2023-11-24T08:44:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1654957056 bytes 2023-11-24T08:44:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1765689344 bytes 2023-11-24T08:44:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1766297600 bytes 2023-11-24T08:44:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1779612672 bytes 2023-11-24T08:44:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1871240192 bytes 2023-11-24T08:44:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1953107968 bytes 2023-11-24T08:44:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 2027064320 bytes 2023-11-24T08:44:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2101051392 bytes 2023-11-24T08:44:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2158984192 bytes 2023-11-24T08:44:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2168201216 bytes 2023-11-24T08:44:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2198774784 bytes 2023-11-24T08:44:32.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2265728000 bytes 2023-11-24T08:44:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2291993600 bytes 2023-11-24T08:44:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2322394112 bytes 2023-11-24T08:44:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2428618752 bytes 2023-11-24T08:44:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2481885184 bytes 2023-11-24T08:44:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:44:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:44:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:44:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_16 2023-11-24T08:44:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-11-24T08:44:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_16 128M 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2023-11-24T08:44:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:44:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:44:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-11-24T08:44:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:44:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-11-24T08:44:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:44:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-11-24T08:44:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_16': 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 130982797c7f 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.130982797c7f 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:33 2023 2023-11-24T08:44:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:33 2023 2023-11-24T08:44:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:33 2023 2023-11-24T08:44:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:44:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:44:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 00787dee-37ed-4352-8685-7ea4d59a610b 2023-11-24T08:44:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:44:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:44:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_16 2023-11-24T08:44:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:44:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:44:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2023-11-24T08:44:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:44:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:44:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 11342848 bytes 2023-11-24T08:44:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 40272896 bytes 2023-11-24T08:44:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 84371456 bytes 2023-11-24T08:44:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 111099904 bytes 2023-11-24T08:44:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 178714624 bytes 2023-11-24T08:44:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 215797760 bytes 2023-11-24T08:44:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 241116160 bytes 2023-11-24T08:44:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 281986048 bytes 2023-11-24T08:44:36.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 369489920 bytes 2023-11-24T08:44:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 434251776 bytes 2023-11-24T08:44:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 454045696 bytes 2023-11-24T08:44:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 486095872 bytes 2023-11-24T08:44:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 550884352 bytes 2023-11-24T08:44:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 579780608 bytes 2023-11-24T08:44:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 580386816 bytes 2023-11-24T08:44:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 635252736 bytes 2023-11-24T08:44:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 703875072 bytes 2023-11-24T08:44:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 738086912 bytes 2023-11-24T08:44:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 756661248 bytes 2023-11-24T08:44:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 847721472 bytes 2023-11-24T08:44:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 867729408 bytes 2023-11-24T08:44:39.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 939965440 bytes 2023-11-24T08:44:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 945513472 bytes 2023-11-24T08:44:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 988611584 bytes 2023-11-24T08:44:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1034398720 bytes 2023-11-24T08:44:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1058556928 bytes 2023-11-24T08:44:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1115669504 bytes 2023-11-24T08:44:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1175154688 bytes 2023-11-24T08:44:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1195803648 bytes 2023-11-24T08:44:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1239067648 bytes 2023-11-24T08:44:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1255752704 bytes 2023-11-24T08:44:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1279905792 bytes 2023-11-24T08:44:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1293497344 bytes 2023-11-24T08:44:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1314385920 bytes 2023-11-24T08:44:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1351538688 bytes 2023-11-24T08:44:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1404657664 bytes 2023-11-24T08:44:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1447438336 bytes 2023-11-24T08:44:41.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1490535424 bytes 2023-11-24T08:44:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1537489920 bytes 2023-11-24T08:44:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1569449984 bytes 2023-11-24T08:44:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1587482624 bytes 2023-11-24T08:44:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1595145216 bytes 2023-11-24T08:44:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1595770880 bytes 2023-11-24T08:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1626755072 bytes 2023-11-24T08:44:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:44:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:44:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:44:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_17 2023-11-24T08:44:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-11-24T08:44:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:44:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:44:42.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_17 128M 2023-11-24T08:44:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:44:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:44:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:44:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:44:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2023-11-24T08:44:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:44:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:44:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-11-24T08:44:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:44:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:42.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-11-24T08:44:43.584 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-24T08:44:43.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-24T08:44:43.584 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-11-24T08:44:43.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T08:44:43.585+0000 7fe5f323a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 47328) UID: 1000 2023-11-24T08:44:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:44:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-11-24T08:44:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_17': 2023-11-24T08:44:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:44:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:44:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:44:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13117f1258b 2023-11-24T08:44:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13117f1258b 2023-11-24T08:44:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:44:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:44:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:44:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:44:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:42 2023 2023-11-24T08:44:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:42 2023 2023-11-24T08:44:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:42 2023 2023-11-24T08:44:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:44:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:44:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 12e91ddf-7a62-43f6-8172-24011e6b5d43 2023-11-24T08:44:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:44:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:44:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_17 2023-11-24T08:44:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:44:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:44:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T08:44:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:44:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:44:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 20736000 bytes 2023-11-24T08:44:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 23326720 bytes 2023-11-24T08:44:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 39296000 bytes 2023-11-24T08:44:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 49114112 bytes 2023-11-24T08:44:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 73360384 bytes 2023-11-24T08:44:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 98972672 bytes 2023-11-24T08:44:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 130219008 bytes 2023-11-24T08:44:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 130844672 bytes 2023-11-24T08:44:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 143382528 bytes 2023-11-24T08:44:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 162111488 bytes 2023-11-24T08:44:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 242138112 bytes 2023-11-24T08:44:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 269208576 bytes 2023-11-24T08:44:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 279987200 bytes 2023-11-24T08:44:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 337040384 bytes 2023-11-24T08:44:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 401666048 bytes 2023-11-24T08:44:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 415767552 bytes 2023-11-24T08:44:46.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 416398336 bytes 2023-11-24T08:44:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 473822208 bytes 2023-11-24T08:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 571190272 bytes 2023-11-24T08:44:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 576593920 bytes 2023-11-24T08:44:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 636519424 bytes 2023-11-24T08:44:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 657657856 bytes 2023-11-24T08:44:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 699422720 bytes 2023-11-24T08:44:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 751234048 bytes 2023-11-24T08:44:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 765672448 bytes 2023-11-24T08:44:48.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 784830464 bytes 2023-11-24T08:44:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 801266688 bytes 2023-11-24T08:44:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 820161536 bytes 2023-11-24T08:44:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 824692736 bytes 2023-11-24T08:44:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 867943424 bytes 2023-11-24T08:44:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 878701568 bytes 2023-11-24T08:44:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 928136192 bytes 2023-11-24T08:44:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 945069056 bytes 2023-11-24T08:44:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 984417280 bytes 2023-11-24T08:44:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1025271808 bytes 2023-11-24T08:44:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1051662336 bytes 2023-11-24T08:44:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1105331200 bytes 2023-11-24T08:44:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1140455424 bytes 2023-11-24T08:44:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1187947520 bytes 2023-11-24T08:44:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1284841472 bytes 2023-11-24T08:44:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1363766272 bytes 2023-11-24T08:44:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1384008704 bytes 2023-11-24T08:44:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1440845824 bytes 2023-11-24T08:44:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1557904384 bytes 2023-11-24T08:44:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1573968896 bytes 2023-11-24T08:44:51.585 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-24T08:44:51.585 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:44:51.586 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-24T08:44:51.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-24T08:44:51.586 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-24T08:44:51.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-24T08:44:51.589 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-11-24T08:44:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1601475584 bytes 2023-11-24T08:44:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1648232448 bytes 2023-11-24T08:44:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1693187072 bytes 2023-11-24T08:44:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1726209024 bytes 2023-11-24T08:44:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1762095104 bytes 2023-11-24T08:44:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1846553600 bytes 2023-11-24T08:44:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1872428032 bytes 2023-11-24T08:44:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1880315904 bytes 2023-11-24T08:44:53.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 1896709120 bytes 2023-11-24T08:44:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 1945125888 bytes 2023-11-24T08:44:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2059552768 bytes 2023-11-24T08:44:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2086649856 bytes 2023-11-24T08:44:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:44:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:44:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:44:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_18 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_18 128M 2023-11-24T08:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2023-11-24T08:44:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:44:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:44:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:44:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-11-24T08:44:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:44:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:44:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-11-24T08:44:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:44:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-11-24T08:44:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_18': 2023-11-24T08:44:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13219a78afc7 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13219a78afc7 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:53 2023 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:53 2023 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:53 2023 2023-11-24T08:44:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:44:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:44:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: de67f348-91f1-4254-a508-55ce6ec8848d 2023-11-24T08:44:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:44:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:44:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_18 2023-11-24T08:44:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:44:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:44:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:44:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:44:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2023-11-24T08:44:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:44:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:44:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 54614016 bytes 2023-11-24T08:44:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 84652032 bytes 2023-11-24T08:44:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 131412992 bytes 2023-11-24T08:44:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 205215744 bytes 2023-11-24T08:44:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 243755008 bytes 2023-11-24T08:44:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 329299968 bytes 2023-11-24T08:44:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 348221440 bytes 2023-11-24T08:44:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 474724352 bytes 2023-11-24T08:44:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 534440960 bytes 2023-11-24T08:44:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 574783488 bytes 2023-11-24T08:44:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 598998016 bytes 2023-11-24T08:44:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 656288768 bytes 2023-11-24T08:44:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 726838272 bytes 2023-11-24T08:44:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 844382208 bytes 2023-11-24T08:44:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 929802240 bytes 2023-11-24T08:44:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 1007744000 bytes 2023-11-24T08:44:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 1030597632 bytes 2023-11-24T08:44:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 1034833920 bytes 2023-11-24T08:44:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1042186240 bytes 2023-11-24T08:44:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1102855168 bytes 2023-11-24T08:44:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1165215744 bytes 2023-11-24T08:44:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1231200256 bytes 2023-11-24T08:44:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1263606784 bytes 2023-11-24T08:44:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1264229376 bytes 2023-11-24T08:44:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1333144576 bytes 2023-11-24T08:44:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1355622400 bytes 2023-11-24T08:44:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1399337984 bytes 2023-11-24T08:44:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1431433216 bytes 2023-11-24T08:45:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1463617536 bytes 2023-11-24T08:45:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1488996352 bytes 2023-11-24T08:45:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1513937920 bytes 2023-11-24T08:45:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1558791168 bytes 2023-11-24T08:45:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1559409664 bytes 2023-11-24T08:45:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1622523904 bytes 2023-11-24T08:45:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1689487360 bytes 2023-11-24T08:45:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1715223552 bytes 2023-11-24T08:45:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1782110208 bytes 2023-11-24T08:45:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1865018368 bytes 2023-11-24T08:45:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1945053184 bytes 2023-11-24T08:45:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 2023230464 bytes 2023-11-24T08:45:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 2110951424 bytes 2023-11-24T08:45:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2145968128 bytes 2023-11-24T08:45:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2164839424 bytes 2023-11-24T08:45:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2178023424 bytes 2023-11-24T08:45:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2242322432 bytes 2023-11-24T08:45:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2265955328 bytes 2023-11-24T08:45:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2290806784 bytes 2023-11-24T08:45:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2313200640 bytes 2023-11-24T08:45:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2316444672 bytes 2023-11-24T08:45:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2340374528 bytes 2023-11-24T08:45:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2353458176 bytes 2023-11-24T08:45:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2366671872 bytes 2023-11-24T08:45:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2396144640 bytes 2023-11-24T08:45:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2454410240 bytes 2023-11-24T08:45:04.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2461549568 bytes 2023-11-24T08:45:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2495706112 bytes 2023-11-24T08:45:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2502970368 bytes 2023-11-24T08:45:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2547360768 bytes 2023-11-24T08:45:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2555463680 bytes 2023-11-24T08:45:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2603441152 bytes 2023-11-24T08:45:05.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2642439168 bytes 2023-11-24T08:45:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2659570688 bytes 2023-11-24T08:45:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2718465024 bytes 2023-11-24T08:45:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2762624000 bytes 2023-11-24T08:45:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2851496960 bytes 2023-11-24T08:45:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2875160576 bytes 2023-11-24T08:45:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2942304256 bytes 2023-11-24T08:45:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3022355456 bytes 2023-11-24T08:45:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3075569664 bytes 2023-11-24T08:45:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3131020288 bytes 2023-11-24T08:45:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3158148096 bytes 2023-11-24T08:45:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3243528192 bytes 2023-11-24T08:45:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3282273280 bytes 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_19 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_19 128M 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:45:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:45:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:45:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2023-11-24T08:45:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:45:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:45:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-11-24T08:45:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:45:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-11-24T08:45:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:45:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-11-24T08:45:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_19': 2023-11-24T08:45:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:45:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:45:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:45:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 132aeaf812a1 2023-11-24T08:45:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.132aeaf812a1 2023-11-24T08:45:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:45:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:45:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:45:08.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:45:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:08 2023 2023-11-24T08:45:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:08 2023 2023-11-24T08:45:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:08 2023 2023-11-24T08:45:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:45:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:45:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 7bc813da-10f2-42ef-b0da-2017c7edc945 2023-11-24T08:45:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:45:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:45:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_19 2023-11-24T08:45:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:45:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:45:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2023-11-24T08:45:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:45:08.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:45:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 81625088 bytes 2023-11-24T08:45:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 98488320 bytes 2023-11-24T08:45:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 163851264 bytes 2023-11-24T08:45:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 200125440 bytes 2023-11-24T08:45:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 259494912 bytes 2023-11-24T08:45:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 309000192 bytes 2023-11-24T08:45:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 358184960 bytes 2023-11-24T08:45:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 438811648 bytes 2023-11-24T08:45:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 484393984 bytes 2023-11-24T08:45:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 515394560 bytes 2023-11-24T08:45:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 554178560 bytes 2023-11-24T08:45:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 638926848 bytes 2023-11-24T08:45:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 639562752 bytes 2023-11-24T08:45:11.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 682394624 bytes 2023-11-24T08:45:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 697664512 bytes 2023-11-24T08:45:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 729995264 bytes 2023-11-24T08:45:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 754081792 bytes 2023-11-24T08:45:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 798457856 bytes 2023-11-24T08:45:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 804249600 bytes 2023-11-24T08:45:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 857800704 bytes 2023-11-24T08:45:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 865080320 bytes 2023-11-24T08:45:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 999712768 bytes 2023-11-24T08:45:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1018437632 bytes 2023-11-24T08:45:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1093150720 bytes 2023-11-24T08:45:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:45:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_20 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_20 128M 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:45:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-11-24T08:45:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:45:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-11-24T08:45:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_20': 2023-11-24T08:45:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:45:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1334419077e3 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1334419077e3 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:13 2023 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:13 2023 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:13 2023 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:45:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9f9af0b3-95c5-474e-831d-a21ecd5dc147 2023-11-24T08:45:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:45:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:45:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_20 2023-11-24T08:45:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:45:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:45:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2023-11-24T08:45:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:45:14.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:45:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 36612096 bytes 2023-11-24T08:45:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 41309184 bytes 2023-11-24T08:45:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 85311488 bytes 2023-11-24T08:45:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 132934656 bytes 2023-11-24T08:45:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 200897536 bytes 2023-11-24T08:45:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 228337664 bytes 2023-11-24T08:45:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 257079296 bytes 2023-11-24T08:45:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 330234880 bytes 2023-11-24T08:45:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 356422656 bytes 2023-11-24T08:45:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 378549248 bytes 2023-11-24T08:45:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 401233920 bytes 2023-11-24T08:45:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 473295872 bytes 2023-11-24T08:45:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 513736704 bytes 2023-11-24T08:45:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 531365888 bytes 2023-11-24T08:45:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 594029568 bytes 2023-11-24T08:45:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 594668544 bytes 2023-11-24T08:45:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 658210816 bytes 2023-11-24T08:45:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 693840896 bytes 2023-11-24T08:45:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 729818112 bytes 2023-11-24T08:45:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 761874432 bytes 2023-11-24T08:45:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 792946688 bytes 2023-11-24T08:45:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 797169664 bytes 2023-11-24T08:45:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 805793792 bytes 2023-11-24T08:45:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 836079616 bytes 2023-11-24T08:45:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 924487680 bytes 2023-11-24T08:45:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 943370240 bytes 2023-11-24T08:45:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 952557568 bytes 2023-11-24T08:45:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 967273472 bytes 2023-11-24T08:45:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1058523136 bytes 2023-11-24T08:45:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1149025280 bytes 2023-11-24T08:45:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1203816448 bytes 2023-11-24T08:45:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1222768640 bytes 2023-11-24T08:45:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1243504640 bytes 2023-11-24T08:45:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1262439424 bytes 2023-11-24T08:45:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1293166592 bytes 2023-11-24T08:45:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1319362560 bytes 2023-11-24T08:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1385590784 bytes 2023-11-24T08:45:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1397972992 bytes 2023-11-24T08:45:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1427944448 bytes 2023-11-24T08:45:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1477072896 bytes 2023-11-24T08:45:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1489925120 bytes 2023-11-24T08:45:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1515478016 bytes 2023-11-24T08:45:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1521864704 bytes 2023-11-24T08:45:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1573791744 bytes 2023-11-24T08:45:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1611825152 bytes 2023-11-24T08:45:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1651160064 bytes 2023-11-24T08:45:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1703987200 bytes 2023-11-24T08:45:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1741380608 bytes 2023-11-24T08:45:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1781220352 bytes 2023-11-24T08:45:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1792101376 bytes 2023-11-24T08:45:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1844332544 bytes 2023-11-24T08:45:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1936905216 bytes 2023-11-24T08:45:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1958411264 bytes 2023-11-24T08:45:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 1973198848 bytes 2023-11-24T08:45:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 1995899904 bytes 2023-11-24T08:45:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2035129344 bytes 2023-11-24T08:45:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2065080320 bytes 2023-11-24T08:45:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2144881664 bytes 2023-11-24T08:45:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2155716608 bytes 2023-11-24T08:45:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2223371264 bytes 2023-11-24T08:45:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2260387840 bytes 2023-11-24T08:45:25.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2306847744 bytes 2023-11-24T08:45:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2330697728 bytes 2023-11-24T08:45:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2369602560 bytes 2023-11-24T08:45:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2376872960 bytes 2023-11-24T08:45:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2440487936 bytes 2023-11-24T08:45:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2455790592 bytes 2023-11-24T08:45:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2506542080 bytes 2023-11-24T08:45:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2562818048 bytes 2023-11-24T08:45:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2606277632 bytes 2023-11-24T08:45:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2689308672 bytes 2023-11-24T08:45:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 2790627328 bytes 2023-11-24T08:45:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 2813831168 bytes 2023-11-24T08:45:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 2895846400 bytes 2023-11-24T08:45:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 2896446464 bytes 2023-11-24T08:45:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3018392576 bytes 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_21 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_21 128M 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:45:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:45:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:45:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:45:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2023-11-24T08:45:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:45:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:45:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-11-24T08:45:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:45:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-11-24T08:45:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:45:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-11-24T08:45:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_21': 2023-11-24T08:45:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:45:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 133ff47131c1 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.133ff47131c1 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:28 2023 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:28 2023 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:28 2023 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:45:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:45:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 3cc6e0f9-790b-43bf-a15d-2b4b68e157bd 2023-11-24T08:45:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:45:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:45:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_21 2023-11-24T08:45:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:45:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:45:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2023-11-24T08:45:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:45:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:45:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 51525632 bytes 2023-11-24T08:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 70962176 bytes 2023-11-24T08:45:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 139106304 bytes 2023-11-24T08:45:30.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 180344832 bytes 2023-11-24T08:45:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 225705984 bytes 2023-11-24T08:45:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 226311168 bytes 2023-11-24T08:45:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 246982656 bytes 2023-11-24T08:45:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 310064128 bytes 2023-11-24T08:45:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 326064128 bytes 2023-11-24T08:45:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 353996800 bytes 2023-11-24T08:45:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 395371520 bytes 2023-11-24T08:45:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 434487296 bytes 2023-11-24T08:45:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 470921216 bytes 2023-11-24T08:45:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 497769472 bytes 2023-11-24T08:45:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 526534656 bytes 2023-11-24T08:45:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 588108800 bytes 2023-11-24T08:45:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 613343232 bytes 2023-11-24T08:45:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 690638848 bytes 2023-11-24T08:45:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 700281856 bytes 2023-11-24T08:45:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 742518784 bytes 2023-11-24T08:45:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 844645376 bytes 2023-11-24T08:45:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 891110400 bytes 2023-11-24T08:45:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 921711616 bytes 2023-11-24T08:45:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 956776448 bytes 2023-11-24T08:45:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 996699136 bytes 2023-11-24T08:45:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1081205760 bytes 2023-11-24T08:45:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1148174336 bytes 2023-11-24T08:45:35.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1217611776 bytes 2023-11-24T08:45:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1268006912 bytes 2023-11-24T08:45:35.480 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6142487289914818) > weight (0.25) 2023-11-24T08:45:35.481 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.49422705525726895) > weight (0.25) 2023-11-24T08:45:35.481 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4468046692186568) > weight (0.25) 2023-11-24T08:45:35.481 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8058133023563069) > weight (0.25) 2023-11-24T08:45:35.481 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-11-24T08:45:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1280502784 bytes 2023-11-24T08:45:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1331322880 bytes 2023-11-24T08:45:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1388306432 bytes 2023-11-24T08:45:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1439453184 bytes 2023-11-24T08:45:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1531628544 bytes 2023-11-24T08:45:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1579610112 bytes 2023-11-24T08:45:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1636745216 bytes 2023-11-24T08:45:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1664621568 bytes 2023-11-24T08:45:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1717702656 bytes 2023-11-24T08:45:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1815234560 bytes 2023-11-24T08:45:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1886668800 bytes 2023-11-24T08:45:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1933677568 bytes 2023-11-24T08:45:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1995327488 bytes 2023-11-24T08:45:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2067115008 bytes 2023-11-24T08:45:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2132249600 bytes 2023-11-24T08:45:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2141635584 bytes 2023-11-24T08:45:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2144025600 bytes 2023-11-24T08:45:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2191649792 bytes 2023-11-24T08:45:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2229401600 bytes 2023-11-24T08:45:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2297908224 bytes 2023-11-24T08:45:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2330181632 bytes 2023-11-24T08:45:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2365097984 bytes 2023-11-24T08:45:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2374789120 bytes 2023-11-24T08:45:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2464878592 bytes 2023-11-24T08:45:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2510349312 bytes 2023-11-24T08:45:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2590578688 bytes 2023-11-24T08:45:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2643644416 bytes 2023-11-24T08:45:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2661169152 bytes 2023-11-24T08:45:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2729223168 bytes 2023-11-24T08:45:41.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2752432128 bytes 2023-11-24T08:45:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2888717312 bytes 2023-11-24T08:45:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2902680576 bytes 2023-11-24T08:45:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2912634880 bytes 2023-11-24T08:45:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2951100416 bytes 2023-11-24T08:45:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2954508288 bytes 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_22 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_22 128M 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:45:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2023-11-24T08:45:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:45:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:45:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-11-24T08:45:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:45:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-11-24T08:45:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:45:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-11-24T08:45:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_22': 2023-11-24T08:45:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:45:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:45:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:45:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1348b98d744f 2023-11-24T08:45:43.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1348b98d744f 2023-11-24T08:45:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:45:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:45:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:45:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:45:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:42 2023 2023-11-24T08:45:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:42 2023 2023-11-24T08:45:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:42 2023 2023-11-24T08:45:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:45:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:45:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 7c3b3b3c-f77d-402e-adec-8115e36a2f88 2023-11-24T08:45:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:45:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:45:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_22 2023-11-24T08:45:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:45:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:45:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2023-11-24T08:45:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:45:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:45:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 83549184 bytes 2023-11-24T08:45:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 107862016 bytes 2023-11-24T08:45:44.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 181727232 bytes 2023-11-24T08:45:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 210417664 bytes 2023-11-24T08:45:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 233893888 bytes 2023-11-24T08:45:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 272537600 bytes 2023-11-24T08:45:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 314413056 bytes 2023-11-24T08:45:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 341839872 bytes 2023-11-24T08:45:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 410397696 bytes 2023-11-24T08:45:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 456323072 bytes 2023-11-24T08:45:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 470918144 bytes 2023-11-24T08:45:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 471524352 bytes 2023-11-24T08:45:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 523763712 bytes 2023-11-24T08:45:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 538110976 bytes 2023-11-24T08:45:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 613995520 bytes 2023-11-24T08:45:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 640573440 bytes 2023-11-24T08:45:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 675803136 bytes 2023-11-24T08:45:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 738501632 bytes 2023-11-24T08:45:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 772438016 bytes 2023-11-24T08:45:47.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 820458496 bytes 2023-11-24T08:45:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 858645504 bytes 2023-11-24T08:45:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 945030144 bytes 2023-11-24T08:45:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 953616384 bytes 2023-11-24T08:45:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 998312960 bytes 2023-11-24T08:45:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1010192384 bytes 2023-11-24T08:45:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1032412160 bytes 2023-11-24T08:45:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1033010176 bytes 2023-11-24T08:45:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1078279168 bytes 2023-11-24T08:45:48.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1121117184 bytes 2023-11-24T08:45:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1138286592 bytes 2023-11-24T08:45:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1182573568 bytes 2023-11-24T08:45:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1229471744 bytes 2023-11-24T08:45:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1243720704 bytes 2023-11-24T08:45:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1287248896 bytes 2023-11-24T08:45:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1388285952 bytes 2023-11-24T08:45:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1439427584 bytes 2023-11-24T08:45:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1482563584 bytes 2023-11-24T08:45:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1497819136 bytes 2023-11-24T08:45:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1504356352 bytes 2023-11-24T08:45:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1542248448 bytes 2023-11-24T08:45:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1554997248 bytes 2023-11-24T08:45:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1630740480 bytes 2023-11-24T08:45:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1717917696 bytes 2023-11-24T08:45:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1800643584 bytes 2023-11-24T08:45:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1802828800 bytes 2023-11-24T08:45:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1829299200 bytes 2023-11-24T08:45:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1829932032 bytes 2023-11-24T08:45:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1874712576 bytes 2023-11-24T08:45:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1958658048 bytes 2023-11-24T08:45:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2002714624 bytes 2023-11-24T08:45:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2027167744 bytes 2023-11-24T08:45:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2129604608 bytes 2023-11-24T08:45:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2212685824 bytes 2023-11-24T08:45:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2233967616 bytes 2023-11-24T08:45:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2299806720 bytes 2023-11-24T08:45:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2358076416 bytes 2023-11-24T08:45:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2370448384 bytes 2023-11-24T08:45:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2406877184 bytes 2023-11-24T08:45:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_23 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_23 128M 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:45:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:45:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:45:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:45:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2023-11-24T08:45:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:45:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:45:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:45:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-11-24T08:45:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:45:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-11-24T08:45:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:45:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-11-24T08:45:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_23': 2023-11-24T08:45:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:45:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:45:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:45:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1351215bd19c 2023-11-24T08:45:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1351215bd19c 2023-11-24T08:45:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:54 2023 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:54 2023 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:54 2023 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 6e18b899-53e6-40ab-b047-1c57f3a06280 2023-11-24T08:45:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:45:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:45:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_23 2023-11-24T08:45:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:45:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:45:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:45:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:45:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2023-11-24T08:45:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:45:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:45:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 50337792 bytes 2023-11-24T08:45:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 92516352 bytes 2023-11-24T08:45:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 115084288 bytes 2023-11-24T08:45:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 182853632 bytes 2023-11-24T08:45:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 213745664 bytes 2023-11-24T08:45:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 240112640 bytes 2023-11-24T08:45:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 247921664 bytes 2023-11-24T08:45:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 256770048 bytes 2023-11-24T08:45:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 306495488 bytes 2023-11-24T08:45:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 361465856 bytes 2023-11-24T08:45:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 389562368 bytes 2023-11-24T08:45:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 390174720 bytes 2023-11-24T08:45:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 391563264 bytes 2023-11-24T08:45:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 434665472 bytes 2023-11-24T08:45:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 489606144 bytes 2023-11-24T08:45:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 549556224 bytes 2023-11-24T08:45:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 577027072 bytes 2023-11-24T08:45:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 577650688 bytes 2023-11-24T08:45:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 634764288 bytes 2023-11-24T08:45:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 646160384 bytes 2023-11-24T08:45:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 685090816 bytes 2023-11-24T08:45:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 728222720 bytes 2023-11-24T08:45:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 776706048 bytes 2023-11-24T08:45:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 917390336 bytes 2023-11-24T08:46:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1009389568 bytes 2023-11-24T08:46:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1074118656 bytes 2023-11-24T08:46:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1083028480 bytes 2023-11-24T08:46:00.590 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.49461144298565096) > weight (0.25) 2023-11-24T08:46:00.590 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9651314889785796) > weight (0.25) 2023-11-24T08:46:00.590 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.40266282842292844) > weight (0.25) 2023-11-24T08:46:00.590 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-11-24T08:46:00.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-11-24T08:46:00.591 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-24T08:46:00.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:46:00.592+0000 7f45f8c15640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 39814) UID: 1000 2023-11-24T08:46:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1119754240 bytes 2023-11-24T08:46:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1152093184 bytes 2023-11-24T08:46:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1165815808 bytes 2023-11-24T08:46:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1270312960 bytes 2023-11-24T08:46:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1308298240 bytes 2023-11-24T08:46:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1361420288 bytes 2023-11-24T08:46:01.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1371299840 bytes 2023-11-24T08:46:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1449658368 bytes 2023-11-24T08:46:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1516630016 bytes 2023-11-24T08:46:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1540654080 bytes 2023-11-24T08:46:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1617129472 bytes 2023-11-24T08:46:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1695113216 bytes 2023-11-24T08:46:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1707998208 bytes 2023-11-24T08:46:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1733219328 bytes 2023-11-24T08:46:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1760580608 bytes 2023-11-24T08:46:03.592 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-11-24T08:46:03.592 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:46:03.592 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-11-24T08:46:03.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-11-24T08:46:03.593 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-11-24T08:46:03.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-11-24T08:46:03.596 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-11-24T08:46:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1869105152 bytes 2023-11-24T08:46:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1918176256 bytes 2023-11-24T08:46:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1948439552 bytes 2023-11-24T08:46:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1983895552 bytes 2023-11-24T08:46:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2046075904 bytes 2023-11-24T08:46:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2059490304 bytes 2023-11-24T08:46:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2108663808 bytes 2023-11-24T08:46:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2129201152 bytes 2023-11-24T08:46:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2160751616 bytes 2023-11-24T08:46:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2229223424 bytes 2023-11-24T08:46:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2277792768 bytes 2023-11-24T08:46:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2306106368 bytes 2023-11-24T08:46:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2409167872 bytes 2023-11-24T08:46:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2527171584 bytes 2023-11-24T08:46:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2539804672 bytes 2023-11-24T08:46:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2553515008 bytes 2023-11-24T08:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2571785216 bytes 2023-11-24T08:46:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2632611840 bytes 2023-11-24T08:46:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:46:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:46:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:46:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_24 2023-11-24T08:46:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-11-24T08:46:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:46:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:07.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_24 128M 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:46:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:46:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:46:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2023-11-24T08:46:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:46:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:46:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-11-24T08:46:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:46:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-11-24T08:46:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:46:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-11-24T08:46:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_24': 2023-11-24T08:46:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:46:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:46:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:46:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 135f4c84f7c1 2023-11-24T08:46:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.135f4c84f7c1 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:07 2023 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:07 2023 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:07 2023 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 2238b317-dd07-4472-9062-bd252e8143ec 2023-11-24T08:46:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:46:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:46:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_24 2023-11-24T08:46:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:46:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:46:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2023-11-24T08:46:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:46:08.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:46:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 71428096 bytes 2023-11-24T08:46:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 111119360 bytes 2023-11-24T08:46:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 162625536 bytes 2023-11-24T08:46:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 183190528 bytes 2023-11-24T08:46:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 219285504 bytes 2023-11-24T08:46:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 250926080 bytes 2023-11-24T08:46:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 256412672 bytes 2023-11-24T08:46:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 270437376 bytes 2023-11-24T08:46:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 329477120 bytes 2023-11-24T08:46:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 384562176 bytes 2023-11-24T08:46:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 438642688 bytes 2023-11-24T08:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 561500160 bytes 2023-11-24T08:46:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 612749312 bytes 2023-11-24T08:46:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 637516800 bytes 2023-11-24T08:46:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 685490176 bytes 2023-11-24T08:46:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 717552640 bytes 2023-11-24T08:46:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 771670016 bytes 2023-11-24T08:46:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 829949952 bytes 2023-11-24T08:46:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 860119040 bytes 2023-11-24T08:46:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 874921984 bytes 2023-11-24T08:46:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 878449664 bytes 2023-11-24T08:46:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 935205888 bytes 2023-11-24T08:46:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1080212480 bytes 2023-11-24T08:46:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1109932032 bytes 2023-11-24T08:46:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1120209920 bytes 2023-11-24T08:46:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:46:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:46:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:46:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_25 2023-11-24T08:46:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_25 128M 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:46:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:46:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:46:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:46:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2023-11-24T08:46:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:46:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:46:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-11-24T08:46:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:46:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-11-24T08:46:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:46:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-11-24T08:46:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_25': 2023-11-24T08:46:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:46:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:46:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:46:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1367a8592e9a 2023-11-24T08:46:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1367a8592e9a 2023-11-24T08:46:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:46:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:46:14.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:46:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:46:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:13 2023 2023-11-24T08:46:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:13 2023 2023-11-24T08:46:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:13 2023 2023-11-24T08:46:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:46:14.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:46:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9114b26e-4897-4df7-936d-4ec836b81477 2023-11-24T08:46:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:46:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:46:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_25 2023-11-24T08:46:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:46:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:46:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2023-11-24T08:46:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:46:14.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:46:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 3222528 bytes 2023-11-24T08:46:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 41874432 bytes 2023-11-24T08:46:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 58641408 bytes 2023-11-24T08:46:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 69511168 bytes 2023-11-24T08:46:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 149014528 bytes 2023-11-24T08:46:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 165449728 bytes 2023-11-24T08:46:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 179721216 bytes 2023-11-24T08:46:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 217094144 bytes 2023-11-24T08:46:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 258327552 bytes 2023-11-24T08:46:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 290418688 bytes 2023-11-24T08:46:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 311954432 bytes 2023-11-24T08:46:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 360116224 bytes 2023-11-24T08:46:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 422795264 bytes 2023-11-24T08:46:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 456169472 bytes 2023-11-24T08:46:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 516832256 bytes 2023-11-24T08:46:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 580914176 bytes 2023-11-24T08:46:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 597920768 bytes 2023-11-24T08:46:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 623504384 bytes 2023-11-24T08:46:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 673061888 bytes 2023-11-24T08:46:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 689006592 bytes 2023-11-24T08:46:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 709033984 bytes 2023-11-24T08:46:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 757725184 bytes 2023-11-24T08:46:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 791876608 bytes 2023-11-24T08:46:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 813221888 bytes 2023-11-24T08:46:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 871534592 bytes 2023-11-24T08:46:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 910823424 bytes 2023-11-24T08:46:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 921804800 bytes 2023-11-24T08:46:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 923235328 bytes 2023-11-24T08:46:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 945786880 bytes 2023-11-24T08:46:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 993948672 bytes 2023-11-24T08:46:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1033384960 bytes 2023-11-24T08:46:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1117221888 bytes 2023-11-24T08:46:20.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1127487488 bytes 2023-11-24T08:46:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1144472576 bytes 2023-11-24T08:46:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1223735296 bytes 2023-11-24T08:46:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1292203008 bytes 2023-11-24T08:46:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1331503104 bytes 2023-11-24T08:46:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1344451584 bytes 2023-11-24T08:46:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1366678528 bytes 2023-11-24T08:46:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1422314496 bytes 2023-11-24T08:46:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1434115072 bytes 2023-11-24T08:46:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1471559680 bytes 2023-11-24T08:46:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1560152064 bytes 2023-11-24T08:46:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1607331840 bytes 2023-11-24T08:46:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1679315968 bytes 2023-11-24T08:46:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1703726080 bytes 2023-11-24T08:46:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1772978176 bytes 2023-11-24T08:46:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1826811904 bytes 2023-11-24T08:46:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1837376512 bytes 2023-11-24T08:46:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:46:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:46:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:46:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_26 2023-11-24T08:46:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-11-24T08:46:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_26 128M 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:46:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:46:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:46:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:46:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:23.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2023-11-24T08:46:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:46:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:46:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-11-24T08:46:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:46:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-11-24T08:46:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:46:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-11-24T08:46:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_26': 2023-11-24T08:46:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:46:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:46:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:46:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13705cf3f93e 2023-11-24T08:46:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13705cf3f93e 2023-11-24T08:46:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:46:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:46:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:46:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:46:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:23 2023 2023-11-24T08:46:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:23 2023 2023-11-24T08:46:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:23 2023 2023-11-24T08:46:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:46:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:46:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: e79a8072-8f71-4674-9961-be59755f749a 2023-11-24T08:46:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:46:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:46:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_26 2023-11-24T08:46:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:46:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:46:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2023-11-24T08:46:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:46:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:46:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 40195072 bytes 2023-11-24T08:46:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 48909312 bytes 2023-11-24T08:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 80525312 bytes 2023-11-24T08:46:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 127889408 bytes 2023-11-24T08:46:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 225513472 bytes 2023-11-24T08:46:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 252590080 bytes 2023-11-24T08:46:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 302292992 bytes 2023-11-24T08:46:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 368380928 bytes 2023-11-24T08:46:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 386510848 bytes 2023-11-24T08:46:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 476662784 bytes 2023-11-24T08:46:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 509547520 bytes 2023-11-24T08:46:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 568934400 bytes 2023-11-24T08:46:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 602421248 bytes 2023-11-24T08:46:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 617645056 bytes 2023-11-24T08:46:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 660355072 bytes 2023-11-24T08:46:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 675459072 bytes 2023-11-24T08:46:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 716155904 bytes 2023-11-24T08:46:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 740914176 bytes 2023-11-24T08:46:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 775294976 bytes 2023-11-24T08:46:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 807887872 bytes 2023-11-24T08:46:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 861592576 bytes 2023-11-24T08:46:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 964600832 bytes 2023-11-24T08:46:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 996190208 bytes 2023-11-24T08:46:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1013142528 bytes 2023-11-24T08:46:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1059904512 bytes 2023-11-24T08:46:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1139289088 bytes 2023-11-24T08:46:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1206410240 bytes 2023-11-24T08:46:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1234592768 bytes 2023-11-24T08:46:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1268971520 bytes 2023-11-24T08:46:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1309458432 bytes 2023-11-24T08:46:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1376078848 bytes 2023-11-24T08:46:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1401843712 bytes 2023-11-24T08:46:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1436480512 bytes 2023-11-24T08:46:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1505276928 bytes 2023-11-24T08:46:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1535073280 bytes 2023-11-24T08:46:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1541944320 bytes 2023-11-24T08:46:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1647082496 bytes 2023-11-24T08:46:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1647702016 bytes 2023-11-24T08:46:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1694196736 bytes 2023-11-24T08:46:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1718444032 bytes 2023-11-24T08:46:32.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1752019968 bytes 2023-11-24T08:46:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1771182080 bytes 2023-11-24T08:46:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1783208960 bytes 2023-11-24T08:46:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1875054592 bytes 2023-11-24T08:46:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1918594048 bytes 2023-11-24T08:46:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1992476672 bytes 2023-11-24T08:46:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2004241408 bytes 2023-11-24T08:46:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2038114304 bytes 2023-11-24T08:46:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2111015936 bytes 2023-11-24T08:46:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2139669504 bytes 2023-11-24T08:46:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2143126528 bytes 2023-11-24T08:46:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2184408064 bytes 2023-11-24T08:46:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2185808896 bytes 2023-11-24T08:46:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2211810304 bytes 2023-11-24T08:46:34.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2214515712 bytes 2023-11-24T08:46:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2249063424 bytes 2023-11-24T08:46:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2301884416 bytes 2023-11-24T08:46:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_27 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_27 128M 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:46:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:46:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:46:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2023-11-24T08:46:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:46:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:46:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-11-24T08:46:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:46:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-11-24T08:46:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:46:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-11-24T08:46:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_27': 2023-11-24T08:46:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:46:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:46:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1378e6a4a021 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1378e6a4a021 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:35 2023 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:35 2023 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:35 2023 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:46:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 170b38bb-5513-4ade-9ac7-4a9823217a98 2023-11-24T08:46:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:46:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:46:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_27 2023-11-24T08:46:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:46:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:46:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T08:46:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:46:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:46:36.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 46402560 bytes 2023-11-24T08:46:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 121432064 bytes 2023-11-24T08:46:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 156884992 bytes 2023-11-24T08:46:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 279559168 bytes 2023-11-24T08:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 372753408 bytes 2023-11-24T08:46:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 397478912 bytes 2023-11-24T08:46:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 432326656 bytes 2023-11-24T08:46:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 483993600 bytes 2023-11-24T08:46:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 553989120 bytes 2023-11-24T08:46:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 569968640 bytes 2023-11-24T08:46:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 629882880 bytes 2023-11-24T08:46:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 669824000 bytes 2023-11-24T08:46:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 707672064 bytes 2023-11-24T08:46:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 757004288 bytes 2023-11-24T08:46:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 766606336 bytes 2023-11-24T08:46:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 789167104 bytes 2023-11-24T08:46:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 846694400 bytes 2023-11-24T08:46:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 886063104 bytes 2023-11-24T08:46:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 948056064 bytes 2023-11-24T08:46:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 968409088 bytes 2023-11-24T08:46:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 969052160 bytes 2023-11-24T08:46:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 981971968 bytes 2023-11-24T08:46:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1033282560 bytes 2023-11-24T08:46:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1070269440 bytes 2023-11-24T08:46:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1101036544 bytes 2023-11-24T08:46:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1158992896 bytes 2023-11-24T08:46:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1183669248 bytes 2023-11-24T08:46:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1248849920 bytes 2023-11-24T08:46:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1284146176 bytes 2023-11-24T08:46:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1290166272 bytes 2023-11-24T08:46:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1332646912 bytes 2023-11-24T08:46:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1353162752 bytes 2023-11-24T08:46:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1417333760 bytes 2023-11-24T08:46:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1450737664 bytes 2023-11-24T08:46:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1482913792 bytes 2023-11-24T08:46:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1523563520 bytes 2023-11-24T08:46:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1589311488 bytes 2023-11-24T08:46:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1621610496 bytes 2023-11-24T08:46:43.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1656045568 bytes 2023-11-24T08:46:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1747686400 bytes 2023-11-24T08:46:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1784218624 bytes 2023-11-24T08:46:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1891998720 bytes 2023-11-24T08:46:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1912243200 bytes 2023-11-24T08:46:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1943752704 bytes 2023-11-24T08:46:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1978676224 bytes 2023-11-24T08:46:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2021679104 bytes 2023-11-24T08:46:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2095279104 bytes 2023-11-24T08:46:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2180035584 bytes 2023-11-24T08:46:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2202889216 bytes 2023-11-24T08:46:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:46:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:46:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_28 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:46:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_28 128M 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-11-24T08:46:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:46:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_28': 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1380f0b4fb25 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1380f0b4fb25 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:46:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:46:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:46 2023 2023-11-24T08:46:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:46 2023 2023-11-24T08:46:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:46 2023 2023-11-24T08:46:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:46:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:46:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: dd54726b-38b5-4534-a5b9-b85acdcdf924 2023-11-24T08:46:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:46:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:46:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_28 2023-11-24T08:46:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:46:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:46:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2023-11-24T08:46:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:46:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:46:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 11346944 bytes 2023-11-24T08:46:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 11963392 bytes 2023-11-24T08:46:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 65609728 bytes 2023-11-24T08:46:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 82064384 bytes 2023-11-24T08:46:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 143642624 bytes 2023-11-24T08:46:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 209666048 bytes 2023-11-24T08:46:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 235272192 bytes 2023-11-24T08:46:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 299211776 bytes 2023-11-24T08:46:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 356638720 bytes 2023-11-24T08:46:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 447950848 bytes 2023-11-24T08:46:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 504102912 bytes 2023-11-24T08:46:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 525945856 bytes 2023-11-24T08:46:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 564185088 bytes 2023-11-24T08:46:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 592392192 bytes 2023-11-24T08:46:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 609933312 bytes 2023-11-24T08:46:49.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 688509952 bytes 2023-11-24T08:46:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 696036352 bytes 2023-11-24T08:46:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 738720768 bytes 2023-11-24T08:46:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 746054656 bytes 2023-11-24T08:46:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 779758592 bytes 2023-11-24T08:46:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 817794048 bytes 2023-11-24T08:46:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 851281920 bytes 2023-11-24T08:46:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 906191872 bytes 2023-11-24T08:46:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 934359040 bytes 2023-11-24T08:46:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1027788800 bytes 2023-11-24T08:46:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1072830464 bytes 2023-11-24T08:46:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1156667392 bytes 2023-11-24T08:46:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1169430528 bytes 2023-11-24T08:46:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1240014848 bytes 2023-11-24T08:46:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1248446464 bytes 2023-11-24T08:46:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1291973632 bytes 2023-11-24T08:46:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:46:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_29 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_29 128M 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:46:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:46:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:46:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2023-11-24T08:46:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:46:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:46:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:46:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-11-24T08:46:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:46:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:46:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-11-24T08:46:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:46:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-11-24T08:46:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_29': 2023-11-24T08:46:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:46:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 138a696dfc67 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.138a696dfc67 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:52 2023 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:52 2023 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:52 2023 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:46:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9ed4d512-1473-4b48-9cd4-c90c2f60808e 2023-11-24T08:46:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:46:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:46:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_29 2023-11-24T08:46:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:46:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:46:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:46:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:46:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=12 2023-11-24T08:46:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:46:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:46:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 46323712 bytes 2023-11-24T08:46:53.482 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7305689002019413) > weight (0.25) 2023-11-24T08:46:53.482 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5277277603237186) > weight (0.25) 2023-11-24T08:46:53.482 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-11-24T08:46:53.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-11-24T08:46:53.483 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-24T08:46:53.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T08:46:53.484+0000 7f565408b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 39822) UID: 1000 2023-11-24T08:46:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 127233024 bytes 2023-11-24T08:46:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 170021888 bytes 2023-11-24T08:46:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 221111296 bytes 2023-11-24T08:46:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 268078080 bytes 2023-11-24T08:46:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 331318272 bytes 2023-11-24T08:46:54.484 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-11-24T08:46:54.484 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:46:54.484 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-11-24T08:46:54.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-11-24T08:46:54.484 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-11-24T08:46:54.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-11-24T08:46:54.487 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-11-24T08:46:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 336939008 bytes 2023-11-24T08:46:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 354462720 bytes 2023-11-24T08:46:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 398015488 bytes 2023-11-24T08:46:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 425786368 bytes 2023-11-24T08:46:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 482034688 bytes 2023-11-24T08:46:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 557445120 bytes 2023-11-24T08:46:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 560096256 bytes 2023-11-24T08:46:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 586415104 bytes 2023-11-24T08:46:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 671341568 bytes 2023-11-24T08:46:56.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 671949824 bytes 2023-11-24T08:46:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 785394688 bytes 2023-11-24T08:46:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 808466432 bytes 2023-11-24T08:46:57.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 872224768 bytes 2023-11-24T08:46:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 942534656 bytes 2023-11-24T08:46:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 984119296 bytes 2023-11-24T08:46:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1045693440 bytes 2023-11-24T08:46:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1063379968 bytes 2023-11-24T08:46:58.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1098182656 bytes 2023-11-24T08:46:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1124681728 bytes 2023-11-24T08:46:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1130695680 bytes 2023-11-24T08:46:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1158299648 bytes 2023-11-24T08:46:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1162331136 bytes 2023-11-24T08:46:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1220965376 bytes 2023-11-24T08:46:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1292891136 bytes 2023-11-24T08:46:59.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1353655296 bytes 2023-11-24T08:46:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1414312960 bytes 2023-11-24T08:46:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1414921216 bytes 2023-11-24T08:46:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1434967040 bytes 2023-11-24T08:47:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1506767872 bytes 2023-11-24T08:47:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1545539584 bytes 2023-11-24T08:47:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1587398656 bytes 2023-11-24T08:47:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1636026368 bytes 2023-11-24T08:47:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1674303488 bytes 2023-11-24T08:47:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1736701952 bytes 2023-11-24T08:47:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1766831104 bytes 2023-11-24T08:47:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1862860800 bytes 2023-11-24T08:47:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1919825920 bytes 2023-11-24T08:47:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1966448640 bytes 2023-11-24T08:47:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2025803776 bytes 2023-11-24T08:47:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2079702016 bytes 2023-11-24T08:47:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2090253312 bytes 2023-11-24T08:47:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2154871808 bytes 2023-11-24T08:47:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2160263168 bytes 2023-11-24T08:47:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2271935488 bytes 2023-11-24T08:47:03.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2282165248 bytes 2023-11-24T08:47:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2314161152 bytes 2023-11-24T08:47:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2362647552 bytes 2023-11-24T08:47:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2405107712 bytes 2023-11-24T08:47:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2426192896 bytes 2023-11-24T08:47:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2471810048 bytes 2023-11-24T08:47:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2502712320 bytes 2023-11-24T08:47:04.597 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7968277608669215) > weight (0.25) 2023-11-24T08:47:04.597 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-24T08:47:04.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-24T08:47:04.598 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-11-24T08:47:04.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T08:47:04.595+0000 7f5b1e1fa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 44818) UID: 1000 2023-11-24T08:47:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2549871616 bytes 2023-11-24T08:47:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:47:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:47:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:47:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_30 2023-11-24T08:47:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-11-24T08:47:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:47:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:47:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_30 128M 2023-11-24T08:47:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:05.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:47:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:47:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:47:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:47:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2023-11-24T08:47:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:47:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:47:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-11-24T08:47:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:47:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-11-24T08:47:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:47:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-11-24T08:47:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_30': 2023-11-24T08:47:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:47:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 139cdfde95ad 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.139cdfde95ad 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:05 2023 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:05 2023 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:05 2023 2023-11-24T08:47:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:47:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:47:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d77de178-524d-4cb8-ba9c-8399123b0719 2023-11-24T08:47:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:47:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:47:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_30 2023-11-24T08:47:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:47:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:47:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=5 2023-11-24T08:47:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:47:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:47:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 6490112 bytes 2023-11-24T08:47:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 49958912 bytes 2023-11-24T08:47:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 78374912 bytes 2023-11-24T08:47:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 95319040 bytes 2023-11-24T08:47:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 176704512 bytes 2023-11-24T08:47:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 236630016 bytes 2023-11-24T08:47:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 251675648 bytes 2023-11-24T08:47:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 307950592 bytes 2023-11-24T08:47:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 330996736 bytes 2023-11-24T08:47:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 374970368 bytes 2023-11-24T08:47:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 419730432 bytes 2023-11-24T08:47:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 438950912 bytes 2023-11-24T08:47:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 486388736 bytes 2023-11-24T08:47:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 508276736 bytes 2023-11-24T08:47:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 568314880 bytes 2023-11-24T08:47:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 580353024 bytes 2023-11-24T08:47:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 650811392 bytes 2023-11-24T08:47:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 658456576 bytes 2023-11-24T08:47:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 675918848 bytes 2023-11-24T08:47:09.598 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-24T08:47:09.598 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:47:09.599 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-24T08:47:09.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-24T08:47:09.599 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-24T08:47:09.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-24T08:47:09.602 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2023-11-24T08:47:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 696702976 bytes 2023-11-24T08:47:09.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 738782208 bytes 2023-11-24T08:47:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 805336064 bytes 2023-11-24T08:47:10.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 875004928 bytes 2023-11-24T08:47:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 899745792 bytes 2023-11-24T08:47:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 952653824 bytes 2023-11-24T08:47:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 989710336 bytes 2023-11-24T08:47:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:47:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:47:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:47:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_31 2023-11-24T08:47:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-11-24T08:47:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_31 128M 2023-11-24T08:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:47:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:47:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:47:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:47:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2023-11-24T08:47:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:47:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:47:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-11-24T08:47:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:47:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-11-24T08:47:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:47:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_31': 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13adb26c98c5 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13adb26c98c5 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:47:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:11 2023 2023-11-24T08:47:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:11 2023 2023-11-24T08:47:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:11 2023 2023-11-24T08:47:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:47:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:47:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 736a6ea1-a614-4fbc-8e6f-f5a28a8d570f 2023-11-24T08:47:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:47:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:47:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_31 2023-11-24T08:47:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:47:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:47:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2023-11-24T08:47:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:47:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:47:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 79696896 bytes 2023-11-24T08:47:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 114395136 bytes 2023-11-24T08:47:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 159803392 bytes 2023-11-24T08:47:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 230611968 bytes 2023-11-24T08:47:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 265725952 bytes 2023-11-24T08:47:13.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 296259584 bytes 2023-11-24T08:47:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 397042688 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 423052288 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 463237120 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 534194176 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 549416960 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 601520128 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 672654336 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 730698752 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 795691008 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 827003904 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 867181568 bytes 2023-11-24T08:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 926431232 bytes 2023-11-24T08:47:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 950724608 bytes 2023-11-24T08:47:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1025206272 bytes 2023-11-24T08:47:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1061514240 bytes 2023-11-24T08:47:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1100531712 bytes 2023-11-24T08:47:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1118842880 bytes 2023-11-24T08:47:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1159515136 bytes 2023-11-24T08:47:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1197477888 bytes 2023-11-24T08:47:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1264521216 bytes 2023-11-24T08:47:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1273444352 bytes 2023-11-24T08:47:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1295209472 bytes 2023-11-24T08:47:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1347721216 bytes 2023-11-24T08:47:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1380470784 bytes 2023-11-24T08:47:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1402455040 bytes 2023-11-24T08:47:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1419856896 bytes 2023-11-24T08:47:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1443365888 bytes 2023-11-24T08:47:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1517405184 bytes 2023-11-24T08:47:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1574386688 bytes 2023-11-24T08:47:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1595664384 bytes 2023-11-24T08:47:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1618769920 bytes 2023-11-24T08:47:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1627865088 bytes 2023-11-24T08:47:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1714528256 bytes 2023-11-24T08:47:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1765372928 bytes 2023-11-24T08:47:20.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1835090944 bytes 2023-11-24T08:47:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1940221952 bytes 2023-11-24T08:47:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1951043584 bytes 2023-11-24T08:47:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2014598144 bytes 2023-11-24T08:47:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2079518720 bytes 2023-11-24T08:47:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2200280064 bytes 2023-11-24T08:47:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2220848128 bytes 2023-11-24T08:47:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2243596288 bytes 2023-11-24T08:47:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2314992640 bytes 2023-11-24T08:47:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2338275328 bytes 2023-11-24T08:47:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2353847296 bytes 2023-11-24T08:47:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2379346944 bytes 2023-11-24T08:47:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2404652032 bytes 2023-11-24T08:47:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2425064448 bytes 2023-11-24T08:47:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2457107456 bytes 2023-11-24T08:47:23.115 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T08:47:23.111+0000 7fceaa5c3640 -1 reset not still connected to 0x55eabad65110 2023-11-24T08:47:23.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2485476352 bytes 2023-11-24T08:47:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2518674432 bytes 2023-11-24T08:47:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2585857024 bytes 2023-11-24T08:47:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2602603520 bytes 2023-11-24T08:47:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2627866624 bytes 2023-11-24T08:47:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2649022464 bytes 2023-11-24T08:47:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2672109568 bytes 2023-11-24T08:47:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2686241792 bytes 2023-11-24T08:47:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2730266624 bytes 2023-11-24T08:47:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2767852544 bytes 2023-11-24T08:47:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2795400192 bytes 2023-11-24T08:47:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:47:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:47:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:47:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_32 2023-11-24T08:47:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-11-24T08:47:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_32 128M 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:47:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:47:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:47:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:47:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2023-11-24T08:47:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:47:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:47:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-11-24T08:47:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:47:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-11-24T08:47:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:47:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-11-24T08:47:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_32': 2023-11-24T08:47:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:47:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:47:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:47:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13b9fcdde816 2023-11-24T08:47:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13b9fcdde816 2023-11-24T08:47:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:47:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:47:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:47:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:47:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:25 2023 2023-11-24T08:47:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:25 2023 2023-11-24T08:47:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:25 2023 2023-11-24T08:47:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:47:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:47:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: fea5c78b-14c3-43da-93a2-d38278c1541a 2023-11-24T08:47:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:47:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:47:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_32 2023-11-24T08:47:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:47:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:47:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2023-11-24T08:47:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:47:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:47:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 40778752 bytes 2023-11-24T08:47:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 96108544 bytes 2023-11-24T08:47:26.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 169952256 bytes 2023-11-24T08:47:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 183953408 bytes 2023-11-24T08:47:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 224606208 bytes 2023-11-24T08:47:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 275256320 bytes 2023-11-24T08:47:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 323885056 bytes 2023-11-24T08:47:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 355792896 bytes 2023-11-24T08:47:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 356391936 bytes 2023-11-24T08:47:28.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 396793856 bytes 2023-11-24T08:47:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 398742528 bytes 2023-11-24T08:47:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 440574976 bytes 2023-11-24T08:47:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 459954176 bytes 2023-11-24T08:47:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 521607168 bytes 2023-11-24T08:47:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 547805184 bytes 2023-11-24T08:47:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 584963072 bytes 2023-11-24T08:47:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 603203584 bytes 2023-11-24T08:47:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 621554688 bytes 2023-11-24T08:47:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 666469376 bytes 2023-11-24T08:47:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 705293312 bytes 2023-11-24T08:47:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 731821056 bytes 2023-11-24T08:47:30.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 833925120 bytes 2023-11-24T08:47:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 886459392 bytes 2023-11-24T08:47:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 941302784 bytes 2023-11-24T08:47:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 994511872 bytes 2023-11-24T08:47:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1029970944 bytes 2023-11-24T08:47:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1053168640 bytes 2023-11-24T08:47:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1161953280 bytes 2023-11-24T08:47:31.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1207473152 bytes 2023-11-24T08:47:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1210091520 bytes 2023-11-24T08:47:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1251876864 bytes 2023-11-24T08:47:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1309494272 bytes 2023-11-24T08:47:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1372976128 bytes 2023-11-24T08:47:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1456062464 bytes 2023-11-24T08:47:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1500841984 bytes 2023-11-24T08:47:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1529735168 bytes 2023-11-24T08:47:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1545228288 bytes 2023-11-24T08:47:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1632038912 bytes 2023-11-24T08:47:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1668020224 bytes 2023-11-24T08:47:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1712118784 bytes 2023-11-24T08:47:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1764929536 bytes 2023-11-24T08:47:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1806850048 bytes 2023-11-24T08:47:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1863370752 bytes 2023-11-24T08:47:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1935465472 bytes 2023-11-24T08:47:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1994757120 bytes 2023-11-24T08:47:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2026761216 bytes 2023-11-24T08:47:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2073683968 bytes 2023-11-24T08:47:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2100984832 bytes 2023-11-24T08:47:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2119439360 bytes 2023-11-24T08:47:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2143788032 bytes 2023-11-24T08:47:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2166675456 bytes 2023-11-24T08:47:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2237671424 bytes 2023-11-24T08:47:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2293190656 bytes 2023-11-24T08:47:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2397048832 bytes 2023-11-24T08:47:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2431227904 bytes 2023-11-24T08:47:37.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2465611776 bytes 2023-11-24T08:47:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2529808384 bytes 2023-11-24T08:47:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2578736128 bytes 2023-11-24T08:47:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2591347712 bytes 2023-11-24T08:47:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2685765632 bytes 2023-11-24T08:47:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2799250432 bytes 2023-11-24T08:47:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2894489600 bytes 2023-11-24T08:47:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2933809152 bytes 2023-11-24T08:47:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2956594176 bytes 2023-11-24T08:47:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2957222912 bytes 2023-11-24T08:47:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 3000206336 bytes 2023-11-24T08:47:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 3045348352 bytes 2023-11-24T08:47:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 3103464448 bytes 2023-11-24T08:47:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_33 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_33 128M 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:47:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:47:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:47:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:40.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2023-11-24T08:47:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:47:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:47:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-11-24T08:47:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:47:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:40.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-11-24T08:47:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:47:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-11-24T08:47:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_33': 2023-11-24T08:47:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:47:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:47:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:47:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13c566f0cd84 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13c566f0cd84 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:40 2023 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:40 2023 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:40 2023 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:47:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:47:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 6956a2e8-c6e1-4b1f-8188-481ec5f6a81b 2023-11-24T08:47:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:47:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:47:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_33 2023-11-24T08:47:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:47:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:47:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2023-11-24T08:47:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:47:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:47:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 74296320 bytes 2023-11-24T08:47:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 111843328 bytes 2023-11-24T08:47:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 154930176 bytes 2023-11-24T08:47:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 176983040 bytes 2023-11-24T08:47:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 220111872 bytes 2023-11-24T08:47:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 274264064 bytes 2023-11-24T08:47:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 317365248 bytes 2023-11-24T08:47:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 373571584 bytes 2023-11-24T08:47:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 428673024 bytes 2023-11-24T08:47:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 468621312 bytes 2023-11-24T08:47:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 550800384 bytes 2023-11-24T08:47:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 587049984 bytes 2023-11-24T08:47:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 616893440 bytes 2023-11-24T08:47:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 669965312 bytes 2023-11-24T08:47:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 714779648 bytes 2023-11-24T08:47:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 756098048 bytes 2023-11-24T08:47:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 779332608 bytes 2023-11-24T08:47:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 795945984 bytes 2023-11-24T08:47:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 875317248 bytes 2023-11-24T08:47:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 893760512 bytes 2023-11-24T08:47:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 912350208 bytes 2023-11-24T08:47:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 944270336 bytes 2023-11-24T08:47:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 968622080 bytes 2023-11-24T08:47:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1009182720 bytes 2023-11-24T08:47:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1012035584 bytes 2023-11-24T08:47:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1031755776 bytes 2023-11-24T08:47:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1043801088 bytes 2023-11-24T08:47:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1065358336 bytes 2023-11-24T08:47:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1121413120 bytes 2023-11-24T08:47:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1197745152 bytes 2023-11-24T08:47:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1233669120 bytes 2023-11-24T08:47:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1286857728 bytes 2023-11-24T08:47:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1366542336 bytes 2023-11-24T08:47:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1389775872 bytes 2023-11-24T08:47:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1392248832 bytes 2023-11-24T08:47:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1427288064 bytes 2023-11-24T08:47:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1473134592 bytes 2023-11-24T08:47:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1516742656 bytes 2023-11-24T08:47:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1558401024 bytes 2023-11-24T08:47:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1576229888 bytes 2023-11-24T08:47:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1601307648 bytes 2023-11-24T08:47:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1625628672 bytes 2023-11-24T08:47:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1705247744 bytes 2023-11-24T08:47:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1728646144 bytes 2023-11-24T08:47:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1798604800 bytes 2023-11-24T08:47:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1860058112 bytes 2023-11-24T08:47:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1882889216 bytes 2023-11-24T08:47:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1970663424 bytes 2023-11-24T08:47:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2064001024 bytes 2023-11-24T08:47:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2139447296 bytes 2023-11-24T08:47:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2177240064 bytes 2023-11-24T08:47:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2255858688 bytes 2023-11-24T08:47:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2375938048 bytes 2023-11-24T08:47:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2448165888 bytes 2023-11-24T08:47:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2500682752 bytes 2023-11-24T08:47:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2524230656 bytes 2023-11-24T08:47:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2584091648 bytes 2023-11-24T08:47:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2661927936 bytes 2023-11-24T08:47:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2705327104 bytes 2023-11-24T08:47:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2738423808 bytes 2023-11-24T08:47:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2751944704 bytes 2023-11-24T08:47:53.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2780178432 bytes 2023-11-24T08:47:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2780789760 bytes 2023-11-24T08:47:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2841046016 bytes 2023-11-24T08:47:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2922733568 bytes 2023-11-24T08:47:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2945897472 bytes 2023-11-24T08:47:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2975051776 bytes 2023-11-24T08:47:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2978696192 bytes 2023-11-24T08:47:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2983918592 bytes 2023-11-24T08:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:47:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_34 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_34 128M 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:47:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:47:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:47:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-11-24T08:47:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:47:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-11-24T08:47:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_34': 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13d14f93b89b 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13d14f93b89b 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:55 2023 2023-11-24T08:47:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:55 2023 2023-11-24T08:47:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:55 2023 2023-11-24T08:47:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:47:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:47:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 3d5be2fc-e83f-4e85-8d39-4fe13e555474 2023-11-24T08:47:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:47:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:47:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_34 2023-11-24T08:47:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:47:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:47:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:47:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:47:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=13 2023-11-24T08:47:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:47:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:47:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 32405504 bytes 2023-11-24T08:47:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 72900608 bytes 2023-11-24T08:47:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 105145344 bytes 2023-11-24T08:47:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 169719808 bytes 2023-11-24T08:47:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 227277824 bytes 2023-11-24T08:47:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 293423104 bytes 2023-11-24T08:47:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 322271232 bytes 2023-11-24T08:47:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 329932800 bytes 2023-11-24T08:47:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 390662144 bytes 2023-11-24T08:47:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 414234624 bytes 2023-11-24T08:47:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 511593472 bytes 2023-11-24T08:47:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 573562880 bytes 2023-11-24T08:47:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 600514560 bytes 2023-11-24T08:47:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 633143296 bytes 2023-11-24T08:47:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 672973824 bytes 2023-11-24T08:47:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 730878976 bytes 2023-11-24T08:47:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 782256128 bytes 2023-11-24T08:47:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 799206400 bytes 2023-11-24T08:47:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 811043840 bytes 2023-11-24T08:48:00.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 910231552 bytes 2023-11-24T08:48:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 910825472 bytes 2023-11-24T08:48:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 933518336 bytes 2023-11-24T08:48:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 958091264 bytes 2023-11-24T08:48:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1013932032 bytes 2023-11-24T08:48:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1054702592 bytes 2023-11-24T08:48:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1104972800 bytes 2023-11-24T08:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1155895296 bytes 2023-11-24T08:48:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1168835584 bytes 2023-11-24T08:48:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1218538496 bytes 2023-11-24T08:48:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1265422336 bytes 2023-11-24T08:48:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1305469952 bytes 2023-11-24T08:48:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1330161664 bytes 2023-11-24T08:48:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1389383680 bytes 2023-11-24T08:48:02.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1428160512 bytes 2023-11-24T08:48:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1467045888 bytes 2023-11-24T08:48:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1511179264 bytes 2023-11-24T08:48:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1576188928 bytes 2023-11-24T08:48:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1652285440 bytes 2023-11-24T08:48:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1686130688 bytes 2023-11-24T08:48:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1709960192 bytes 2023-11-24T08:48:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1780235264 bytes 2023-11-24T08:48:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1819124736 bytes 2023-11-24T08:48:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1844572160 bytes 2023-11-24T08:48:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1849366528 bytes 2023-11-24T08:48:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1926904832 bytes 2023-11-24T08:48:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1948954624 bytes 2023-11-24T08:48:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1977974784 bytes 2023-11-24T08:48:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2003210240 bytes 2023-11-24T08:48:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2040905728 bytes 2023-11-24T08:48:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2103087104 bytes 2023-11-24T08:48:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2121917440 bytes 2023-11-24T08:48:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2158702592 bytes 2023-11-24T08:48:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2196510720 bytes 2023-11-24T08:48:06.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2243698688 bytes 2023-11-24T08:48:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2292696064 bytes 2023-11-24T08:48:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2327525376 bytes 2023-11-24T08:48:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2366067712 bytes 2023-11-24T08:48:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2407859200 bytes 2023-11-24T08:48:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2431529984 bytes 2023-11-24T08:48:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2441819136 bytes 2023-11-24T08:48:08.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2465828864 bytes 2023-11-24T08:48:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2477860864 bytes 2023-11-24T08:48:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2493579264 bytes 2023-11-24T08:48:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2532853760 bytes 2023-11-24T08:48:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2599502848 bytes 2023-11-24T08:48:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2609907712 bytes 2023-11-24T08:48:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2610509824 bytes 2023-11-24T08:48:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:48:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:48:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:48:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_35 2023-11-24T08:48:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-11-24T08:48:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:48:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_35 128M 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:48:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-11-24T08:48:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:48:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-11-24T08:48:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_35': 2023-11-24T08:48:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:48:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:48:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:48:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13ddc4c9e7fe 2023-11-24T08:48:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13ddc4c9e7fe 2023-11-24T08:48:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:48:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:48:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:09 2023 2023-11-24T08:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:09 2023 2023-11-24T08:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:09 2023 2023-11-24T08:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: c8458320-3838-4775-b00c-eeaf0f63bb66 2023-11-24T08:48:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:48:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:48:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_35 2023-11-24T08:48:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:48:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:48:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2023-11-24T08:48:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:48:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:48:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 47057920 bytes 2023-11-24T08:48:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 82150400 bytes 2023-11-24T08:48:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 107275264 bytes 2023-11-24T08:48:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 184177664 bytes 2023-11-24T08:48:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 251290624 bytes 2023-11-24T08:48:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 274095104 bytes 2023-11-24T08:48:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 296239104 bytes 2023-11-24T08:48:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 304411648 bytes 2023-11-24T08:48:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 371724288 bytes 2023-11-24T08:48:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 383332352 bytes 2023-11-24T08:48:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 411242496 bytes 2023-11-24T08:48:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 411846656 bytes 2023-11-24T08:48:11.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 443586560 bytes 2023-11-24T08:48:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 444201984 bytes 2023-11-24T08:48:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 509638656 bytes 2023-11-24T08:48:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 521752576 bytes 2023-11-24T08:48:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 573932544 bytes 2023-11-24T08:48:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 592623616 bytes 2023-11-24T08:48:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 682755072 bytes 2023-11-24T08:48:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 704490496 bytes 2023-11-24T08:48:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 784708608 bytes 2023-11-24T08:48:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 785312768 bytes 2023-11-24T08:48:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 798704640 bytes 2023-11-24T08:48:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 854607872 bytes 2023-11-24T08:48:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 884971520 bytes 2023-11-24T08:48:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 908270592 bytes 2023-11-24T08:48:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 941710336 bytes 2023-11-24T08:48:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 996345856 bytes 2023-11-24T08:48:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1032244224 bytes 2023-11-24T08:48:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1120103424 bytes 2023-11-24T08:48:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1127666688 bytes 2023-11-24T08:48:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1156489216 bytes 2023-11-24T08:48:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1157117952 bytes 2023-11-24T08:48:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1193826304 bytes 2023-11-24T08:48:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1236305920 bytes 2023-11-24T08:48:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1291435008 bytes 2023-11-24T08:48:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1316811776 bytes 2023-11-24T08:48:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1449587712 bytes 2023-11-24T08:48:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1465511936 bytes 2023-11-24T08:48:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1524012032 bytes 2023-11-24T08:48:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1552250880 bytes 2023-11-24T08:48:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1573530624 bytes 2023-11-24T08:48:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1618960384 bytes 2023-11-24T08:48:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1658031104 bytes 2023-11-24T08:48:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1728117760 bytes 2023-11-24T08:48:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1751079936 bytes 2023-11-24T08:48:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1811225600 bytes 2023-11-24T08:48:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1811848192 bytes 2023-11-24T08:48:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1879904256 bytes 2023-11-24T08:48:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1895589888 bytes 2023-11-24T08:48:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1928199168 bytes 2023-11-24T08:48:19.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 1955594240 bytes 2023-11-24T08:48:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 1984924672 bytes 2023-11-24T08:48:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2004595712 bytes 2023-11-24T08:48:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2091602944 bytes 2023-11-24T08:48:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2165741568 bytes 2023-11-24T08:48:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2249881600 bytes 2023-11-24T08:48:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2329391104 bytes 2023-11-24T08:48:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2362645504 bytes 2023-11-24T08:48:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2410070016 bytes 2023-11-24T08:48:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2486775808 bytes 2023-11-24T08:48:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2500919296 bytes 2023-11-24T08:48:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2532564992 bytes 2023-11-24T08:48:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2562847744 bytes 2023-11-24T08:48:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2617313280 bytes 2023-11-24T08:48:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2679348224 bytes 2023-11-24T08:48:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2721117184 bytes 2023-11-24T08:48:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2728540160 bytes 2023-11-24T08:48:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2741636096 bytes 2023-11-24T08:48:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2772995072 bytes 2023-11-24T08:48:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 2793937920 bytes 2023-11-24T08:48:23.488 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6405946435719584) > weight (0.25) 2023-11-24T08:48:23.488 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2954144631878224) > weight (0.25) 2023-11-24T08:48:23.488 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2939209442367998) > weight (0.25) 2023-11-24T08:48:23.489 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-24T08:48:23.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-24T08:48:23.489 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-11-24T08:48:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T08:48:23.490+0000 7ffaedfd2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 48576) UID: 1000 2023-11-24T08:48:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:48:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:48:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:48:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_36 2023-11-24T08:48:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-11-24T08:48:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:48:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:48:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_36 128M 2023-11-24T08:48:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:48:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:48:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:48:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:48:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2023-11-24T08:48:23.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3639181548282048) > weight (0.25) 2023-11-24T08:48:23.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5059499999757664) > weight (0.25) 2023-11-24T08:48:23.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4083462609955132) > weight (0.25) 2023-11-24T08:48:23.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8370609272528221) > weight (0.25) 2023-11-24T08:48:23.604 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-11-24T08:48:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:48:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:48:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-11-24T08:48:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:48:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-11-24T08:48:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-11-24T08:48:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_36': 2023-11-24T08:48:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:48:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:48:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:48:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13e9951f8488 2023-11-24T08:48:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13e9951f8488 2023-11-24T08:48:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:48:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:48:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:48:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:48:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:23 2023 2023-11-24T08:48:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:23 2023 2023-11-24T08:48:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:23 2023 2023-11-24T08:48:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:48:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:48:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: fc86b320-d291-4bfa-a035-067b30e073fa 2023-11-24T08:48:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:48:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:48:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_36 2023-11-24T08:48:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:48:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:48:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T08:48:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:48:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:48:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 120761344 bytes 2023-11-24T08:48:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 165661696 bytes 2023-11-24T08:48:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 214724608 bytes 2023-11-24T08:48:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 248922112 bytes 2023-11-24T08:48:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 254952448 bytes 2023-11-24T08:48:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 259943424 bytes 2023-11-24T08:48:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 301985792 bytes 2023-11-24T08:48:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 302624768 bytes 2023-11-24T08:48:26.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 316975104 bytes 2023-11-24T08:48:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 327321600 bytes 2023-11-24T08:48:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 364044288 bytes 2023-11-24T08:48:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 468346880 bytes 2023-11-24T08:48:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 510213120 bytes 2023-11-24T08:48:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 576714752 bytes 2023-11-24T08:48:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 579117056 bytes 2023-11-24T08:48:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 604813312 bytes 2023-11-24T08:48:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 674675712 bytes 2023-11-24T08:48:28.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 706470912 bytes 2023-11-24T08:48:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 759027712 bytes 2023-11-24T08:48:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 784449536 bytes 2023-11-24T08:48:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 817502208 bytes 2023-11-24T08:48:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 823001088 bytes 2023-11-24T08:48:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 873250816 bytes 2023-11-24T08:48:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 920480768 bytes 2023-11-24T08:48:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 951307264 bytes 2023-11-24T08:48:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 977776640 bytes 2023-11-24T08:48:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 998402048 bytes 2023-11-24T08:48:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1052122112 bytes 2023-11-24T08:48:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1071574016 bytes 2023-11-24T08:48:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1141816320 bytes 2023-11-24T08:48:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1231144960 bytes 2023-11-24T08:48:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1261257728 bytes 2023-11-24T08:48:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1294743552 bytes 2023-11-24T08:48:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1316339712 bytes 2023-11-24T08:48:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1350360064 bytes 2023-11-24T08:48:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1351211008 bytes 2023-11-24T08:48:31.490 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-24T08:48:31.490 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:48:31.490 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-24T08:48:31.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-24T08:48:31.491 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-24T08:48:31.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-24T08:48:31.494 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-11-24T08:48:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:48:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:48:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:48:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_37 2023-11-24T08:48:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-11-24T08:48:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:48:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:48:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_37 128M 2023-11-24T08:48:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:48:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:48:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:48:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:48:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2023-11-24T08:48:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:48:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:48:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-11-24T08:48:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:48:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-11-24T08:48:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:48:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-11-24T08:48:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_37': 2023-11-24T08:48:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:48:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:48:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:48:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 13f57db887f6 2023-11-24T08:48:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.13f57db887f6 2023-11-24T08:48:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:31 2023 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:31 2023 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:31 2023 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: cb1bf2ef-bc24-41f4-86ea-0b0c7b483a22 2023-11-24T08:48:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:48:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:48:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_37 2023-11-24T08:48:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:48:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:48:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T08:48:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:48:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:48:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 79807488 bytes 2023-11-24T08:48:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 160547840 bytes 2023-11-24T08:48:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 201455616 bytes 2023-11-24T08:48:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 228605952 bytes 2023-11-24T08:48:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 296545280 bytes 2023-11-24T08:48:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 362340352 bytes 2023-11-24T08:48:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 380983296 bytes 2023-11-24T08:48:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 441331712 bytes 2023-11-24T08:48:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 450363392 bytes 2023-11-24T08:48:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 519437312 bytes 2023-11-24T08:48:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 620896256 bytes 2023-11-24T08:48:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 693468160 bytes 2023-11-24T08:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 715262976 bytes 2023-11-24T08:48:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 771796992 bytes 2023-11-24T08:48:35.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 797902848 bytes 2023-11-24T08:48:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 832980992 bytes 2023-11-24T08:48:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 861474816 bytes 2023-11-24T08:48:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 891234304 bytes 2023-11-24T08:48:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 919270400 bytes 2023-11-24T08:48:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 933778432 bytes 2023-11-24T08:48:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1036449792 bytes 2023-11-24T08:48:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1087196160 bytes 2023-11-24T08:48:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1131362304 bytes 2023-11-24T08:48:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1193585664 bytes 2023-11-24T08:48:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1274437632 bytes 2023-11-24T08:48:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1388063744 bytes 2023-11-24T08:48:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1416060928 bytes 2023-11-24T08:48:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1479008256 bytes 2023-11-24T08:48:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1524636672 bytes 2023-11-24T08:48:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1557591040 bytes 2023-11-24T08:48:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1594583040 bytes 2023-11-24T08:48:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:48:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:48:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:48:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_38 2023-11-24T08:48:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-11-24T08:48:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_38 128M 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:48:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:48:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:48:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:48:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-11-24T08:48:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:48:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-11-24T08:48:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_38': 2023-11-24T08:48:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:48:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:48:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:48:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 140394951ca 2023-11-24T08:48:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.140394951ca 2023-11-24T08:48:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:48:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:48:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:48:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:48:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:39 2023 2023-11-24T08:48:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:39 2023 2023-11-24T08:48:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:39 2023 2023-11-24T08:48:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:48:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:48:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 3b2fce77-b38d-48ea-b5c5-cb78f30c2412 2023-11-24T08:48:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:48:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:48:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_38 2023-11-24T08:48:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:48:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:48:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2023-11-24T08:48:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:48:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:48:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 54127616 bytes 2023-11-24T08:48:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 103002112 bytes 2023-11-24T08:48:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 169773056 bytes 2023-11-24T08:48:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 253934592 bytes 2023-11-24T08:48:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 286167040 bytes 2023-11-24T08:48:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 332982272 bytes 2023-11-24T08:48:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 353799168 bytes 2023-11-24T08:48:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 398889984 bytes 2023-11-24T08:48:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 422846464 bytes 2023-11-24T08:48:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 423440384 bytes 2023-11-24T08:48:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 463660032 bytes 2023-11-24T08:48:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 515488768 bytes 2023-11-24T08:48:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 597150720 bytes 2023-11-24T08:48:42.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 633459712 bytes 2023-11-24T08:48:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 650074112 bytes 2023-11-24T08:48:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 689852416 bytes 2023-11-24T08:48:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 773610496 bytes 2023-11-24T08:48:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 823232512 bytes 2023-11-24T08:48:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 831913984 bytes 2023-11-24T08:48:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 900040704 bytes 2023-11-24T08:48:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 926465024 bytes 2023-11-24T08:48:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 978268160 bytes 2023-11-24T08:48:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1034122240 bytes 2023-11-24T08:48:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1115359232 bytes 2023-11-24T08:48:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1164848128 bytes 2023-11-24T08:48:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1217636352 bytes 2023-11-24T08:48:45.014 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:48:45.014+0000 7f42b0626640 -1 reset not still connected to 0x561a56d4cc30 2023-11-24T08:48:45.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1281480704 bytes 2023-11-24T08:48:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1326169088 bytes 2023-11-24T08:48:45.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1384700928 bytes 2023-11-24T08:48:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1440414720 bytes 2023-11-24T08:48:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1504461824 bytes 2023-11-24T08:48:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1526752256 bytes 2023-11-24T08:48:46.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1564308480 bytes 2023-11-24T08:48:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1587938304 bytes 2023-11-24T08:48:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1624682496 bytes 2023-11-24T08:48:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1663589376 bytes 2023-11-24T08:48:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1671656448 bytes 2023-11-24T08:48:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1758025728 bytes 2023-11-24T08:48:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1810672640 bytes 2023-11-24T08:48:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1834418176 bytes 2023-11-24T08:48:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1860289536 bytes 2023-11-24T08:48:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1935241216 bytes 2023-11-24T08:48:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2019152896 bytes 2023-11-24T08:48:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2028702720 bytes 2023-11-24T08:48:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2054178816 bytes 2023-11-24T08:48:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2092043264 bytes 2023-11-24T08:48:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2094905344 bytes 2023-11-24T08:48:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2124765184 bytes 2023-11-24T08:48:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2155928576 bytes 2023-11-24T08:48:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2227353600 bytes 2023-11-24T08:48:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2263567360 bytes 2023-11-24T08:48:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2264155136 bytes 2023-11-24T08:48:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2292772864 bytes 2023-11-24T08:48:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2308016128 bytes 2023-11-24T08:48:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_39 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_39 128M 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:48:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2023-11-24T08:48:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:48:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:48:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-11-24T08:48:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:48:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-11-24T08:48:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:48:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-11-24T08:48:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_39': 2023-11-24T08:48:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:48:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:48:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:48:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 140c27b61a3c 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.140c27b61a3c 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:50 2023 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:50 2023 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:50 2023 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 344f7f3d-a4fa-452d-94e2-e36d500bd26a 2023-11-24T08:48:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:48:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:48:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_39 2023-11-24T08:48:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:48:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:48:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T08:48:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:48:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:48:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 16883712 bytes 2023-11-24T08:48:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 88565760 bytes 2023-11-24T08:48:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 100700160 bytes 2023-11-24T08:48:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 180026368 bytes 2023-11-24T08:48:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 232625152 bytes 2023-11-24T08:48:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 278307840 bytes 2023-11-24T08:48:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 323805184 bytes 2023-11-24T08:48:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 324423680 bytes 2023-11-24T08:48:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 339837952 bytes 2023-11-24T08:48:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 353061888 bytes 2023-11-24T08:48:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 433363968 bytes 2023-11-24T08:48:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 483922944 bytes 2023-11-24T08:48:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 512186368 bytes 2023-11-24T08:48:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 545467392 bytes 2023-11-24T08:48:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 546098176 bytes 2023-11-24T08:48:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 579651584 bytes 2023-11-24T08:48:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 582138880 bytes 2023-11-24T08:48:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 594834432 bytes 2023-11-24T08:48:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 662758400 bytes 2023-11-24T08:48:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 695284736 bytes 2023-11-24T08:48:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 720339968 bytes 2023-11-24T08:48:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 730317824 bytes 2023-11-24T08:48:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 771646464 bytes 2023-11-24T08:48:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 811476992 bytes 2023-11-24T08:48:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 844036096 bytes 2023-11-24T08:48:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 846910464 bytes 2023-11-24T08:48:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 859433984 bytes 2023-11-24T08:48:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 898451456 bytes 2023-11-24T08:48:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 926964736 bytes 2023-11-24T08:48:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 946329600 bytes 2023-11-24T08:48:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 976841728 bytes 2023-11-24T08:48:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1041747968 bytes 2023-11-24T08:48:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1064025088 bytes 2023-11-24T08:48:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1119094784 bytes 2023-11-24T08:48:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1152440320 bytes 2023-11-24T08:48:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1192530944 bytes 2023-11-24T08:48:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1237585920 bytes 2023-11-24T08:48:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1289332736 bytes 2023-11-24T08:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_40 2023-11-24T08:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-11-24T08:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_40 128M 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:48:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2023-11-24T08:48:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:48:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:48:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:48:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-11-24T08:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:48:58.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-11-24T08:48:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:48:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-11-24T08:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_40': 2023-11-24T08:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1415251ee3ed 2023-11-24T08:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1415251ee3ed 2023-11-24T08:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:48:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:48:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:48:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:48:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:58 2023 2023-11-24T08:48:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:58 2023 2023-11-24T08:48:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:58 2023 2023-11-24T08:48:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:48:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:48:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: b34477ff-894a-40fc-ba47-b95068f79b4c 2023-11-24T08:48:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:48:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:48:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_40 2023-11-24T08:48:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:48:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:48:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:48:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:48:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T08:48:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:48:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:48:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 72684544 bytes 2023-11-24T08:48:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 124034048 bytes 2023-11-24T08:48:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 202278912 bytes 2023-11-24T08:48:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 233291776 bytes 2023-11-24T08:48:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 320030720 bytes 2023-11-24T08:49:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 354859008 bytes 2023-11-24T08:49:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 389913600 bytes 2023-11-24T08:49:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 390532096 bytes 2023-11-24T08:49:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 454397952 bytes 2023-11-24T08:49:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 486746112 bytes 2023-11-24T08:49:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 517359616 bytes 2023-11-24T08:49:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 566225920 bytes 2023-11-24T08:49:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 605099008 bytes 2023-11-24T08:49:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 635052032 bytes 2023-11-24T08:49:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 670093312 bytes 2023-11-24T08:49:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 694935552 bytes 2023-11-24T08:49:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 726668288 bytes 2023-11-24T08:49:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 754396160 bytes 2023-11-24T08:49:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 811107328 bytes 2023-11-24T08:49:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 839648256 bytes 2023-11-24T08:49:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 889678848 bytes 2023-11-24T08:49:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 974102528 bytes 2023-11-24T08:49:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1056027648 bytes 2023-11-24T08:49:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1099277312 bytes 2023-11-24T08:49:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1129141248 bytes 2023-11-24T08:49:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1168358400 bytes 2023-11-24T08:49:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1179260928 bytes 2023-11-24T08:49:04.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1237921792 bytes 2023-11-24T08:49:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1262278656 bytes 2023-11-24T08:49:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1316473856 bytes 2023-11-24T08:49:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1389349888 bytes 2023-11-24T08:49:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1471726592 bytes 2023-11-24T08:49:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1485593600 bytes 2023-11-24T08:49:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:49:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:49:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:49:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_41 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:49:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_41 128M 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:49:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2023-11-24T08:49:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:49:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:49:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-11-24T08:49:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:49:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-11-24T08:49:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:49:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-11-24T08:49:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_41': 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 141e74dcdf64 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.141e74dcdf64 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:49:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:05 2023 2023-11-24T08:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:05 2023 2023-11-24T08:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:05 2023 2023-11-24T08:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 647ef1ae-5a98-459b-b31b-d5aa0404c570 2023-11-24T08:49:06.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:49:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:49:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_41 2023-11-24T08:49:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:49:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:49:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=8 2023-11-24T08:49:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:49:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:49:06.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 24091648 bytes 2023-11-24T08:49:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 34757632 bytes 2023-11-24T08:49:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 40219648 bytes 2023-11-24T08:49:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 65274880 bytes 2023-11-24T08:49:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 81107968 bytes 2023-11-24T08:49:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 125889536 bytes 2023-11-24T08:49:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 207867904 bytes 2023-11-24T08:49:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 224958464 bytes 2023-11-24T08:49:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 278539264 bytes 2023-11-24T08:49:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 364762112 bytes 2023-11-24T08:49:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 421319680 bytes 2023-11-24T08:49:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 443012096 bytes 2023-11-24T08:49:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 471593984 bytes 2023-11-24T08:49:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 498887680 bytes 2023-11-24T08:49:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 542908416 bytes 2023-11-24T08:49:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 618380288 bytes 2023-11-24T08:49:09.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 668452864 bytes 2023-11-24T08:49:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 690890752 bytes 2023-11-24T08:49:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 711350272 bytes 2023-11-24T08:49:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 759009280 bytes 2023-11-24T08:49:10.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 872616960 bytes 2023-11-24T08:49:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 873251840 bytes 2023-11-24T08:49:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 889944064 bytes 2023-11-24T08:49:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 970788864 bytes 2023-11-24T08:49:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1010641920 bytes 2023-11-24T08:49:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1046731776 bytes 2023-11-24T08:49:11.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1103815680 bytes 2023-11-24T08:49:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1139484672 bytes 2023-11-24T08:49:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1161191424 bytes 2023-11-24T08:49:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1215174656 bytes 2023-11-24T08:49:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1296391168 bytes 2023-11-24T08:49:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1305241600 bytes 2023-11-24T08:49:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1333259264 bytes 2023-11-24T08:49:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1351940096 bytes 2023-11-24T08:49:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1433879552 bytes 2023-11-24T08:49:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1479464960 bytes 2023-11-24T08:49:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1576992768 bytes 2023-11-24T08:49:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1618423808 bytes 2023-11-24T08:49:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1677830144 bytes 2023-11-24T08:49:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1678417920 bytes 2023-11-24T08:49:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_42 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_42 128M 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:49:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:49:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2023-11-24T08:49:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:49:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:49:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-11-24T08:49:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:49:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-11-24T08:49:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:49:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-11-24T08:49:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_42': 2023-11-24T08:49:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:49:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:49:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 14288bb9e903 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.14288bb9e903 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:14 2023 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:14 2023 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:14 2023 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d4241692-b271-471f-97e7-4e41ae0eefd4 2023-11-24T08:49:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:49:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:49:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_42 2023-11-24T08:49:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:49:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:49:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2023-11-24T08:49:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:49:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:49:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 28012544 bytes 2023-11-24T08:49:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 75797504 bytes 2023-11-24T08:49:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 134521856 bytes 2023-11-24T08:49:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 168594432 bytes 2023-11-24T08:49:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 206413824 bytes 2023-11-24T08:49:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 263989248 bytes 2023-11-24T08:49:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 328131584 bytes 2023-11-24T08:49:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 344388608 bytes 2023-11-24T08:49:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 378047488 bytes 2023-11-24T08:49:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 469092352 bytes 2023-11-24T08:49:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 520881152 bytes 2023-11-24T08:49:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 556498944 bytes 2023-11-24T08:49:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 575349760 bytes 2023-11-24T08:49:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 665474048 bytes 2023-11-24T08:49:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 728098816 bytes 2023-11-24T08:49:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 738167808 bytes 2023-11-24T08:49:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 819566592 bytes 2023-11-24T08:49:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 856614912 bytes 2023-11-24T08:49:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 900080640 bytes 2023-11-24T08:49:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 920942592 bytes 2023-11-24T08:49:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 986194944 bytes 2023-11-24T08:49:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1010636800 bytes 2023-11-24T08:49:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1060680704 bytes 2023-11-24T08:49:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1109925888 bytes 2023-11-24T08:49:20.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1169317888 bytes 2023-11-24T08:49:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1281425408 bytes 2023-11-24T08:49:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1323081728 bytes 2023-11-24T08:49:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:49:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:49:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:49:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_43 2023-11-24T08:49:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-11-24T08:49:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:49:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:49:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_43 128M 2023-11-24T08:49:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:49:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:49:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:49:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:49:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2023-11-24T08:49:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:49:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:49:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-11-24T08:49:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:49:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-11-24T08:49:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:49:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-11-24T08:49:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_43': 2023-11-24T08:49:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:49:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:49:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:49:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1430da4b9b9a 2023-11-24T08:49:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1430da4b9b9a 2023-11-24T08:49:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:49:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:49:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:49:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:49:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:21 2023 2023-11-24T08:49:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:21 2023 2023-11-24T08:49:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:21 2023 2023-11-24T08:49:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:49:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:49:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: ab341c38-aeae-40e6-8dbf-7e351d9676c7 2023-11-24T08:49:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:49:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:49:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_43 2023-11-24T08:49:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:49:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:49:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2023-11-24T08:49:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:49:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:49:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 43680768 bytes 2023-11-24T08:49:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 70090752 bytes 2023-11-24T08:49:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 81767424 bytes 2023-11-24T08:49:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 155511808 bytes 2023-11-24T08:49:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 169584640 bytes 2023-11-24T08:49:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 259051520 bytes 2023-11-24T08:49:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 270406656 bytes 2023-11-24T08:49:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 353107968 bytes 2023-11-24T08:49:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 376805376 bytes 2023-11-24T08:49:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 394592256 bytes 2023-11-24T08:49:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 513697792 bytes 2023-11-24T08:49:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 560952320 bytes 2023-11-24T08:49:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 619889664 bytes 2023-11-24T08:49:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 663192576 bytes 2023-11-24T08:49:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 726309888 bytes 2023-11-24T08:49:25.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 756113408 bytes 2023-11-24T08:49:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 823448576 bytes 2023-11-24T08:49:26.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 876853248 bytes 2023-11-24T08:49:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 897277952 bytes 2023-11-24T08:49:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 934757376 bytes 2023-11-24T08:49:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1007145984 bytes 2023-11-24T08:49:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1057579008 bytes 2023-11-24T08:49:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1116078080 bytes 2023-11-24T08:49:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1177079808 bytes 2023-11-24T08:49:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1259030528 bytes 2023-11-24T08:49:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1301463040 bytes 2023-11-24T08:49:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1318716416 bytes 2023-11-24T08:49:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1395572736 bytes 2023-11-24T08:49:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1417248768 bytes 2023-11-24T08:49:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1469458432 bytes 2023-11-24T08:49:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1535961088 bytes 2023-11-24T08:49:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1555187712 bytes 2023-11-24T08:49:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1574028288 bytes 2023-11-24T08:49:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1697982464 bytes 2023-11-24T08:49:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1766514688 bytes 2023-11-24T08:49:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1819897856 bytes 2023-11-24T08:49:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1839498240 bytes 2023-11-24T08:49:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1869886464 bytes 2023-11-24T08:49:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1921479680 bytes 2023-11-24T08:49:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1960438784 bytes 2023-11-24T08:49:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 2052802560 bytes 2023-11-24T08:49:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2053903360 bytes 2023-11-24T08:49:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2054514688 bytes 2023-11-24T08:49:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2055141376 bytes 2023-11-24T08:49:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2055747584 bytes 2023-11-24T08:49:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2150363136 bytes 2023-11-24T08:49:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2205298688 bytes 2023-11-24T08:49:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2257392640 bytes 2023-11-24T08:49:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2375859200 bytes 2023-11-24T08:49:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2443511808 bytes 2023-11-24T08:49:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2483093504 bytes 2023-11-24T08:49:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:49:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:49:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:49:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_44 2023-11-24T08:49:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-11-24T08:49:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_44 128M 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:49:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:49:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:49:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:49:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2023-11-24T08:49:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:49:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:49:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-11-24T08:49:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:49:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-11-24T08:49:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:49:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-11-24T08:49:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_44': 2023-11-24T08:49:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:49:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:49:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:49:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 143a60c6c688 2023-11-24T08:49:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.143a60c6c688 2023-11-24T08:49:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:49:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:49:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:49:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:49:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:33 2023 2023-11-24T08:49:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:33 2023 2023-11-24T08:49:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:33 2023 2023-11-24T08:49:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:49:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:49:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 6f744513-92ac-4212-8ca4-f89cc0e36fcf 2023-11-24T08:49:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:49:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:49:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_44 2023-11-24T08:49:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:49:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:49:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T08:49:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:49:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:49:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 16419840 bytes 2023-11-24T08:49:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 87616512 bytes 2023-11-24T08:49:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 120086528 bytes 2023-11-24T08:49:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 121964544 bytes 2023-11-24T08:49:35.054 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T08:49:35.053+0000 7fee2f6de640 -1 reset not still connected to 0x55a94978aa90 2023-11-24T08:49:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 175753216 bytes 2023-11-24T08:49:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 202823680 bytes 2023-11-24T08:49:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 290782208 bytes 2023-11-24T08:49:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 307961856 bytes 2023-11-24T08:49:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 386905088 bytes 2023-11-24T08:49:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 502548480 bytes 2023-11-24T08:49:36.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7017480074761807) > weight (0.25) 2023-11-24T08:49:36.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3254560516477756) > weight (0.25) 2023-11-24T08:49:36.606 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7210512142184132) > weight (0.25) 2023-11-24T08:49:36.607 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-11-24T08:49:36.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-11-24T08:49:36.607 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-24T08:49:36.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:49:36.609+0000 7fb571c3b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 50371) UID: 1000 2023-11-24T08:49:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 536366080 bytes 2023-11-24T08:49:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 572135424 bytes 2023-11-24T08:49:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 625939456 bytes 2023-11-24T08:49:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 701409280 bytes 2023-11-24T08:49:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 756614144 bytes 2023-11-24T08:49:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 770805760 bytes 2023-11-24T08:49:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 848906240 bytes 2023-11-24T08:49:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 894951424 bytes 2023-11-24T08:49:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 920799232 bytes 2023-11-24T08:49:38.607 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-11-24T08:49:38.608 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:49:38.608 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-11-24T08:49:38.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-11-24T08:49:38.608 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-11-24T08:49:38.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-11-24T08:49:38.612 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2023-11-24T08:49:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 948851712 bytes 2023-11-24T08:49:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 982219776 bytes 2023-11-24T08:49:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1002998784 bytes 2023-11-24T08:49:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1008346112 bytes 2023-11-24T08:49:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1035313152 bytes 2023-11-24T08:49:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1062857728 bytes 2023-11-24T08:49:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1098956800 bytes 2023-11-24T08:49:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1160838144 bytes 2023-11-24T08:49:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1216416768 bytes 2023-11-24T08:49:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1223275520 bytes 2023-11-24T08:49:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1250727936 bytes 2023-11-24T08:49:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1302793216 bytes 2023-11-24T08:49:40.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1375529984 bytes 2023-11-24T08:49:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1408072704 bytes 2023-11-24T08:49:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1408709632 bytes 2023-11-24T08:49:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1451285504 bytes 2023-11-24T08:49:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1494360064 bytes 2023-11-24T08:49:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1510179840 bytes 2023-11-24T08:49:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1510968320 bytes 2023-11-24T08:49:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1541011456 bytes 2023-11-24T08:49:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1568930816 bytes 2023-11-24T08:49:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1646807040 bytes 2023-11-24T08:49:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1720001536 bytes 2023-11-24T08:49:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1736499200 bytes 2023-11-24T08:49:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1745824768 bytes 2023-11-24T08:49:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1790158848 bytes 2023-11-24T08:49:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1855303680 bytes 2023-11-24T08:49:43.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1888917504 bytes 2023-11-24T08:49:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1904612352 bytes 2023-11-24T08:49:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1977911296 bytes 2023-11-24T08:49:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1995005952 bytes 2023-11-24T08:49:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:49:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:49:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:49:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_45 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:49:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_45 128M 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2023-11-24T08:49:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:49:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:49:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-11-24T08:49:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:49:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-11-24T08:49:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:49:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-11-24T08:49:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_45': 2023-11-24T08:49:44.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1449cd3e403e 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1449cd3e403e 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:44 2023 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:44 2023 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:44 2023 2023-11-24T08:49:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:49:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:49:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 5b977de2-7224-446c-8040-6e733cf6a84b 2023-11-24T08:49:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:49:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:49:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_45 2023-11-24T08:49:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:49:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:49:44.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2023-11-24T08:49:44.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:49:44.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:49:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 90014720 bytes 2023-11-24T08:49:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 139936768 bytes 2023-11-24T08:49:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 158007296 bytes 2023-11-24T08:49:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 253905920 bytes 2023-11-24T08:49:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 330409984 bytes 2023-11-24T08:49:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 380113920 bytes 2023-11-24T08:49:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 415240192 bytes 2023-11-24T08:49:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 419317760 bytes 2023-11-24T08:49:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 451587072 bytes 2023-11-24T08:49:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 478116864 bytes 2023-11-24T08:49:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 510758912 bytes 2023-11-24T08:49:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 580335616 bytes 2023-11-24T08:49:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 628008960 bytes 2023-11-24T08:49:47.434 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:49:47.432+0000 7f4b5a023640 -1 reset not still connected to 0x560218357930 2023-11-24T08:49:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 662065152 bytes 2023-11-24T08:49:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 662665216 bytes 2023-11-24T08:49:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 671618048 bytes 2023-11-24T08:49:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 722454528 bytes 2023-11-24T08:49:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 756506624 bytes 2023-11-24T08:49:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 784613376 bytes 2023-11-24T08:49:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 809745408 bytes 2023-11-24T08:49:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 896764928 bytes 2023-11-24T08:49:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 921402368 bytes 2023-11-24T08:49:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 950296576 bytes 2023-11-24T08:49:49.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 975996928 bytes 2023-11-24T08:49:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1006696448 bytes 2023-11-24T08:49:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1014329344 bytes 2023-11-24T08:49:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1108440064 bytes 2023-11-24T08:49:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1183653888 bytes 2023-11-24T08:49:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1226007552 bytes 2023-11-24T08:49:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1288643584 bytes 2023-11-24T08:49:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1321911296 bytes 2023-11-24T08:49:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1426376704 bytes 2023-11-24T08:49:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1475598336 bytes 2023-11-24T08:49:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1527845888 bytes 2023-11-24T08:49:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1532093440 bytes 2023-11-24T08:49:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1587636224 bytes 2023-11-24T08:49:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1613319168 bytes 2023-11-24T08:49:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1651336192 bytes 2023-11-24T08:49:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1709665280 bytes 2023-11-24T08:49:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1754773504 bytes 2023-11-24T08:49:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1840917504 bytes 2023-11-24T08:49:53.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1921262592 bytes 2023-11-24T08:49:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1942696960 bytes 2023-11-24T08:49:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1953803264 bytes 2023-11-24T08:49:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2038987776 bytes 2023-11-24T08:49:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2048369664 bytes 2023-11-24T08:49:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2098399232 bytes 2023-11-24T08:49:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2130533376 bytes 2023-11-24T08:49:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2180495360 bytes 2023-11-24T08:49:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2223541248 bytes 2023-11-24T08:49:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2285387776 bytes 2023-11-24T08:49:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2324132864 bytes 2023-11-24T08:49:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2395868160 bytes 2023-11-24T08:49:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2469125120 bytes 2023-11-24T08:49:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2499431424 bytes 2023-11-24T08:49:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2562305024 bytes 2023-11-24T08:49:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2604237824 bytes 2023-11-24T08:49:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2656371712 bytes 2023-11-24T08:49:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2679796736 bytes 2023-11-24T08:49:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2712850432 bytes 2023-11-24T08:49:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2745624576 bytes 2023-11-24T08:49:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2768385024 bytes 2023-11-24T08:49:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2778260480 bytes 2023-11-24T08:49:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2854699008 bytes 2023-11-24T08:49:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2855307264 bytes 2023-11-24T08:49:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2915585024 bytes 2023-11-24T08:49:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2936671232 bytes 2023-11-24T08:49:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2986273792 bytes 2023-11-24T08:49:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:49:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:49:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:49:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_46 2023-11-24T08:49:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-11-24T08:49:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_46 128M 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T08:49:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:49:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:49:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:49:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:49:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2023-11-24T08:49:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:49:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-11-24T08:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T08:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:49:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-11-24T08:49:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:49:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-11-24T08:49:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_46': 2023-11-24T08:49:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:49:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:49:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:49:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1455d2e5d5ed 2023-11-24T08:49:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1455d2e5d5ed 2023-11-24T08:49:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:49:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:49:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:58 2023 2023-11-24T08:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:58 2023 2023-11-24T08:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:58 2023 2023-11-24T08:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:49:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 02bfc8e5-4c3a-4991-aa98-8ab3d6782ef1 2023-11-24T08:49:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:49:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:49:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_46 2023-11-24T08:49:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:49:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:49:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T08:49:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:49:59.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T08:49:59.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:49:59.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:49:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 41903104 bytes 2023-11-24T08:50:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 89851904 bytes 2023-11-24T08:50:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 119110656 bytes 2023-11-24T08:50:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 153079808 bytes 2023-11-24T08:50:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 198311936 bytes 2023-11-24T08:50:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 213289984 bytes 2023-11-24T08:50:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 260770816 bytes 2023-11-24T08:50:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 287148032 bytes 2023-11-24T08:50:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 360824832 bytes 2023-11-24T08:50:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 397941760 bytes 2023-11-24T08:50:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 413317120 bytes 2023-11-24T08:50:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 414683136 bytes 2023-11-24T08:50:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 454719488 bytes 2023-11-24T08:50:02.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 495001600 bytes 2023-11-24T08:50:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 538289152 bytes 2023-11-24T08:50:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 591514624 bytes 2023-11-24T08:50:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 614912000 bytes 2023-11-24T08:50:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 664507392 bytes 2023-11-24T08:50:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 717131776 bytes 2023-11-24T08:50:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 803084288 bytes 2023-11-24T08:50:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 828116992 bytes 2023-11-24T08:50:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 884559872 bytes 2023-11-24T08:50:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 967016448 bytes 2023-11-24T08:50:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1016633344 bytes 2023-11-24T08:50:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1084828672 bytes 2023-11-24T08:50:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1120515072 bytes 2023-11-24T08:50:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1164288000 bytes 2023-11-24T08:50:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1182131200 bytes 2023-11-24T08:50:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1215836160 bytes 2023-11-24T08:50:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1258932224 bytes 2023-11-24T08:50:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1319979008 bytes 2023-11-24T08:50:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1342484480 bytes 2023-11-24T08:50:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1380304896 bytes 2023-11-24T08:50:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1405463552 bytes 2023-11-24T08:50:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1476920320 bytes 2023-11-24T08:50:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1502552064 bytes 2023-11-24T08:50:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1546235904 bytes 2023-11-24T08:50:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1571853312 bytes 2023-11-24T08:50:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1597278208 bytes 2023-11-24T08:50:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1617810432 bytes 2023-11-24T08:50:08.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1704235008 bytes 2023-11-24T08:50:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1751131136 bytes 2023-11-24T08:50:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1815665664 bytes 2023-11-24T08:50:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1879272448 bytes 2023-11-24T08:50:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1944766464 bytes 2023-11-24T08:50:09.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2041893888 bytes 2023-11-24T08:50:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2064337920 bytes 2023-11-24T08:50:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_47 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_47 128M 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:50:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:50:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:50:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-11-24T08:50:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:50:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-11-24T08:50:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_47': 2023-11-24T08:50:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 145d2b2203c9 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.145d2b2203c9 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:50:09 2023 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:50:09 2023 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:50:09 2023 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:50:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:50:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 5023e532-77d0-4a81-9a30-bf38f68789b1 2023-11-24T08:50:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:50:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:50:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_47 2023-11-24T08:50:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T08:50:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:50:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T08:50:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:50:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:50:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 39037952 bytes 2023-11-24T08:50:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 97886208 bytes 2023-11-24T08:50:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 163910656 bytes 2023-11-24T08:50:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 186458112 bytes 2023-11-24T08:50:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 242357248 bytes 2023-11-24T08:50:12.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 311919616 bytes 2023-11-24T08:50:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 353693696 bytes 2023-11-24T08:50:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 369357824 bytes 2023-11-24T08:50:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 381971456 bytes 2023-11-24T08:50:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 443425792 bytes 2023-11-24T08:50:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 482309120 bytes 2023-11-24T08:50:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 516599808 bytes 2023-11-24T08:50:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 555498496 bytes 2023-11-24T08:50:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 573751296 bytes 2023-11-24T08:50:13.495 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6719185174990284) > weight (0.25) 2023-11-24T08:50:13.495 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-24T08:50:13.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-24T08:50:13.496 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-11-24T08:50:13.496 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-24T08:50:13.496 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:50:13.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T08:50:13.496+0000 7f81b55ae640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 49297) UID: 1000 2023-11-24T08:50:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 623857664 bytes 2023-11-24T08:50:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 660798464 bytes 2023-11-24T08:50:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 709396480 bytes 2023-11-24T08:50:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 763161600 bytes 2023-11-24T08:50:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 815718400 bytes 2023-11-24T08:50:14.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 828954624 bytes 2023-11-24T08:50:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 842559488 bytes 2023-11-24T08:50:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 844473344 bytes 2023-11-24T08:50:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 949274624 bytes 2023-11-24T08:50:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 984050688 bytes 2023-11-24T08:50:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1013536768 bytes 2023-11-24T08:50:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1081264128 bytes 2023-11-24T08:50:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1111462912 bytes 2023-11-24T08:50:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1176741888 bytes 2023-11-24T08:50:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1222750208 bytes 2023-11-24T08:50:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1290587136 bytes 2023-11-24T08:50:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1324388352 bytes 2023-11-24T08:50:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1423072256 bytes 2023-11-24T08:50:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1440686080 bytes 2023-11-24T08:50:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1451363328 bytes 2023-11-24T08:50:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1507487744 bytes 2023-11-24T08:50:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1550559232 bytes 2023-11-24T08:50:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1603666944 bytes 2023-11-24T08:50:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1687840768 bytes 2023-11-24T08:50:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1719609344 bytes 2023-11-24T08:50:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1739316224 bytes 2023-11-24T08:50:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1758999552 bytes 2023-11-24T08:50:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1796022272 bytes 2023-11-24T08:50:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1836723200 bytes 2023-11-24T08:50:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1871901696 bytes 2023-11-24T08:50:19.598 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-24T08:50:19.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-24T08:50:19.599 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-24T08:50:19.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-24T08:50:19.601 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2023-11-24T08:50:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1916189696 bytes 2023-11-24T08:50:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1961846784 bytes 2023-11-24T08:50:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2053094400 bytes 2023-11-24T08:50:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2143598592 bytes 2023-11-24T08:50:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_48 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_48 128M 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:50:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2023-11-24T08:50:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:50:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:50:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-11-24T08:50:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T08:50:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:50:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-11-24T08:50:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:50:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-11-24T08:50:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_48': 2023-11-24T08:50:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:50:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:50:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:50:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 146f9c41b18d 2023-11-24T08:50:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.146f9c41b18d 2023-11-24T08:50:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:50:20 2023 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:50:20 2023 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:50:20 2023 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 934c4e48-8c50-49b0-b632-f50d1f739feb 2023-11-24T08:50:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:50:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:50:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_48 2023-11-24T08:50:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T08:50:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:50:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T08:50:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:50:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:50:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 90014720 bytes 2023-11-24T08:50:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 147452928 bytes 2023-11-24T08:50:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 176749568 bytes 2023-11-24T08:50:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 267493376 bytes 2023-11-24T08:50:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 301894656 bytes 2023-11-24T08:50:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 352593920 bytes 2023-11-24T08:50:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 410228736 bytes 2023-11-24T08:50:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 442772480 bytes 2023-11-24T08:50:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 493590528 bytes 2023-11-24T08:50:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 540839936 bytes 2023-11-24T08:50:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 587571200 bytes 2023-11-24T08:50:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 676368384 bytes 2023-11-24T08:50:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 695037952 bytes 2023-11-24T08:50:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 734792704 bytes 2023-11-24T08:50:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 735392768 bytes 2023-11-24T08:50:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 817777664 bytes 2023-11-24T08:50:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 886620160 bytes 2023-11-24T08:50:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 944221184 bytes 2023-11-24T08:50:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 1007689728 bytes 2023-11-24T08:50:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1141604352 bytes 2023-11-24T08:50:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1182582784 bytes 2023-11-24T08:50:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1244355584 bytes 2023-11-24T08:50:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1262938112 bytes 2023-11-24T08:50:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1286781952 bytes 2023-11-24T08:50:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1325651968 bytes 2023-11-24T08:50:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1363851264 bytes 2023-11-24T08:50:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1406252032 bytes 2023-11-24T08:50:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1423208448 bytes 2023-11-24T08:50:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1491173376 bytes 2023-11-24T08:50:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1522460672 bytes 2023-11-24T08:50:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1583842304 bytes 2023-11-24T08:50:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:50:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:50:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:50:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_49 2023-11-24T08:50:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-11-24T08:50:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T08:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_49 128M 2023-11-24T08:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T08:50:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:50:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:50:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:50:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2023-11-24T08:50:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:50:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:50:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-11-24T08:50:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T08:50:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:50:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-11-24T08:50:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:50:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-11-24T08:50:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_49': 2023-11-24T08:50:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:50:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:50:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:50:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1478b3be101e 2023-11-24T08:50:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1478b3be101e 2023-11-24T08:50:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:50:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:50:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:50:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:50:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:50:28 2023 2023-11-24T08:50:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:50:28 2023 2023-11-24T08:50:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:50:28 2023 2023-11-24T08:50:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:50:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:50:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: c2dd7404-8a4d-4ca3-966a-b940cfe221e9 2023-11-24T08:50:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:50:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:50:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_49 2023-11-24T08:50:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T08:50:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:50:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2023-11-24T08:50:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:50:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:50:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 6998016 bytes 2023-11-24T08:50:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 63517696 bytes 2023-11-24T08:50:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 121866240 bytes 2023-11-24T08:50:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 184327168 bytes 2023-11-24T08:50:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 337832960 bytes 2023-11-24T08:50:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 371926016 bytes 2023-11-24T08:50:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 403606528 bytes 2023-11-24T08:50:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 458136576 bytes 2023-11-24T08:50:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 471045120 bytes 2023-11-24T08:50:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 532141056 bytes 2023-11-24T08:50:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 557501440 bytes 2023-11-24T08:50:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 604150784 bytes 2023-11-24T08:50:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 640664576 bytes 2023-11-24T08:50:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 706832384 bytes 2023-11-24T08:50:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 737506304 bytes 2023-11-24T08:50:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 820290560 bytes 2023-11-24T08:50:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 865888256 bytes 2023-11-24T08:50:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 921413632 bytes 2023-11-24T08:50:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 990131200 bytes 2023-11-24T08:50:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 1015849984 bytes 2023-11-24T08:50:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1060404224 bytes 2023-11-24T08:50:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1101199360 bytes 2023-11-24T08:50:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1161399296 bytes 2023-11-24T08:50:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1201113088 bytes 2023-11-24T08:50:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1216412672 bytes 2023-11-24T08:50:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1227028480 bytes 2023-11-24T08:50:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1300980736 bytes 2023-11-24T08:50:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1343896576 bytes 2023-11-24T08:50:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1356343296 bytes 2023-11-24T08:50:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1388786688 bytes 2023-11-24T08:50:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1389399040 bytes 2023-11-24T08:50:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1465762816 bytes 2023-11-24T08:50:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1556961280 bytes 2023-11-24T08:50:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1595128832 bytes 2023-11-24T08:50:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1668593664 bytes 2023-11-24T08:50:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1742861312 bytes 2023-11-24T08:50:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1759931392 bytes 2023-11-24T08:50:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1798103040 bytes 2023-11-24T08:50:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1833443328 bytes 2023-11-24T08:50:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1847800832 bytes 2023-11-24T08:50:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1873001472 bytes 2023-11-24T08:50:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1926502400 bytes 2023-11-24T08:50:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1993416704 bytes 2023-11-24T08:50:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2010263552 bytes 2023-11-24T08:50:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2038307840 bytes 2023-11-24T08:50:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2099471360 bytes 2023-11-24T08:50:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2149003264 bytes 2023-11-24T08:50:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2149602304 bytes 2023-11-24T08:50:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2161166336 bytes 2023-11-24T08:50:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2210132992 bytes 2023-11-24T08:50:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2268605440 bytes 2023-11-24T08:50:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2289449984 bytes 2023-11-24T08:50:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2362164224 bytes 2023-11-24T08:50:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:50:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:50:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:50:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_50 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n snapshot ] 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_50 128M 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 128M ] 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=128M 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:50:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2023-11-24T08:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ is_pool_mirror_mode_image mirror 2023-11-24T08:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ image = image ] 2023-11-24T08:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-11-24T08:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T08:50:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T08:50:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-11-24T08:50:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T08:50:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-11-24T08:50:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_50': 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1482a5e4d8bf 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1482a5e4d8bf 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:50:40 2023 2023-11-24T08:50:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:50:40 2023 2023-11-24T08:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:50:40 2023 2023-11-24T08:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 16bfa33f-6709-47f2-9afa-1242602344ab 2023-11-24T08:50:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T08:50:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n ] 2023-11-24T08:50:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_50 2023-11-24T08:50:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T08:50:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T08:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2023-11-24T08:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T08:50:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-11-24T08:50:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 99916800 bytes 2023-11-24T08:50:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 159870976 bytes 2023-11-24T08:50:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 203952128 bytes 2023-11-24T08:50:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 227212288 bytes 2023-11-24T08:50:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 233485312 bytes 2023-11-24T08:50:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 295146496 bytes 2023-11-24T08:50:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 349892608 bytes 2023-11-24T08:50:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 413421568 bytes 2023-11-24T08:50:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 414036992 bytes 2023-11-24T08:50:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 484444160 bytes 2023-11-24T08:50:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 507922432 bytes 2023-11-24T08:50:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 554248192 bytes 2023-11-24T08:50:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 570480640 bytes 2023-11-24T08:50:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 642462720 bytes 2023-11-24T08:50:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 669967360 bytes 2023-11-24T08:50:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 693126144 bytes 2023-11-24T08:50:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 806078464 bytes 2023-11-24T08:50:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 812709888 bytes 2023-11-24T08:50:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 886835200 bytes 2023-11-24T08:50:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 976319488 bytes 2023-11-24T08:50:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 995434496 bytes 2023-11-24T08:50:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1037927424 bytes 2023-11-24T08:50:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1080184832 bytes 2023-11-24T08:50:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1145496576 bytes 2023-11-24T08:50:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1154865152 bytes 2023-11-24T08:50:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1248312320 bytes 2023-11-24T08:50:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1311088640 bytes 2023-11-24T08:50:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1364625408 bytes 2023-11-24T08:50:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1376081920 bytes 2023-11-24T08:50:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1431115776 bytes 2023-11-24T08:50:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1444619264 bytes 2023-11-24T08:50:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1445279744 bytes 2023-11-24T08:50:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1478317056 bytes 2023-11-24T08:50:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1494863872 bytes 2023-11-24T08:50:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1563194368 bytes 2023-11-24T08:50:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1585958912 bytes 2023-11-24T08:50:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1626679296 bytes 2023-11-24T08:50:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1674461184 bytes 2023-11-24T08:50:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1711495168 bytes 2023-11-24T08:50:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1757591552 bytes 2023-11-24T08:50:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1797513216 bytes 2023-11-24T08:50:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1813222400 bytes 2023-11-24T08:50:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1838270464 bytes 2023-11-24T08:50:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1889077248 bytes 2023-11-24T08:50:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1920979968 bytes 2023-11-24T08:50:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1981459456 bytes 2023-11-24T08:50:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2033392640 bytes 2023-11-24T08:50:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2152622080 bytes 2023-11-24T08:50:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2177012736 bytes 2023-11-24T08:50:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2198092800 bytes 2023-11-24T08:50:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2248181760 bytes 2023-11-24T08:50:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2260542464 bytes 2023-11-24T08:50:52.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2264473600 bytes 2023-11-24T08:50:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_count=50 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local count 2023-11-24T08:50:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_count=0 2023-11-24T08:50:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:50:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 40 2023-11-24T08:50:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2023-11-24T08:50:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd+ cut -d -f 2 2023-11-24T08:50:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep images: 2023-11-24T08:50:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: --cluster cluster2 -p mirror mirror pool status 2023-11-24T08:50:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:50:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count=50 2023-11-24T08:50:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 50 = 50 2023-11-24T08:50:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-11-24T08:50:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:50:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:50:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state 2023-11-24T08:50:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 40 2023-11-24T08:50:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2023-11-24T08:50:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-11-24T08:50:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep image health: 2023-11-24T08:50:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cut -d -f 3 2023-11-24T08:50:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:50:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=OK 2023-11-24T08:50:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = ERROR 2023-11-24T08:50:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = OK 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_count=50 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local count 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_count=0 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 40 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-11-24T08:50:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep images: 2023-11-24T08:50:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cut -d -f 2 2023-11-24T08:50:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:50:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count=50 2023-11-24T08:50:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 50 = 50 2023-11-24T08:50:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:50:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-11-24T08:50:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:50:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:50:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:50:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state 2023-11-24T08:50:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 40 2023-11-24T08:50:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2023-11-24T08:50:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-11-24T08:50:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep image health: 2023-11-24T08:50:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cut -d -f 3 2023-11-24T08:50:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:50:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=WARNING 2023-11-24T08:50:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test WARNING = ERROR 2023-11-24T08:50:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test WARNING = OK 2023-11-24T08:50:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 2 -ne 1 2023-11-24T08:50:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 30 2023-11-24T08:51:15.614 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.46912327708955615) > weight (0.25) 2023-11-24T08:51:15.614 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3312898546959743) > weight (0.25) 2023-11-24T08:51:15.614 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-11-24T08:51:15.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-11-24T08:51:15.615 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-24T08:51:15.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:51:15.611+0000 7fa1ecdaa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 45825) UID: 1000 2023-11-24T08:51:16.615 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-11-24T08:51:16.616 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:51:16.616 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-11-24T08:51:16.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-11-24T08:51:16.616 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-11-24T08:51:16.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-11-24T08:51:16.619 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-11-24T08:51:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + rbd --cluster cluster1 -p mirror mirror poolgrep image health: status 2023-11-24T08:51:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cut -d -f 3 2023-11-24T08:51:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=OK 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = ERROR 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = OK 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog TEST: compare many images 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.tyy3Nh9UuT/rbd-mirror.test.log 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ date +%F %T 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2023-11-24 08:51:22 TEST: compare many images 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2023-11-24 08:51:22 TEST: compare many images 2023-11-24T08:51:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 50 2023-11-24T08:51:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2023-11-24T08:51:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_1 snap 2023-11-24T08:51:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:51:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-11-24T08:51:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-11-24T08:51:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:51:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:51:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-11-24T08:51:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:51:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:51:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:51:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + admin_daemons cluster1 rbd mirrorgrep status -i state mirror/image_1 2023-11-24T08:51:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:51:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:51:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:51:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:51:25.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:51:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-11-24T08:51:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:51:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:25.566 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:51:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-11-24T08:51:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:no valid command found; 1 closest matches: 2023-11-24T08:51:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_18 2023-11-24T08:51:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2023-11-24T08:51:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:51:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:51:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:51:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:51:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:51:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2023-11-24T08:51:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:51:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:51:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:51:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:51:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-11-24T08:51:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_1.status 2023-11-24T08:51:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_1.status 2023-11-24T08:51:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_1.status 2023-11-24T08:51:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_1.status 2023-11-24T08:51:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_1.status 2023-11-24T08:51:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.status 2023-11-24T08:51:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-11-24T08:51:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-11-24T08:51:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 119 2023-11-24T08:51:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_1.status 2023-11-24T08:51:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_1.status 2023-11-24T08:51:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-11-24T08:51:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:51:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=119 2023-11-24T08:51:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:51:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.status 2023-11-24T08:51:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.status 2023-11-24T08:51:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-11-24T08:51:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:51:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:51:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:51:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 119 = 2023-11-24T08:51:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:51:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.status 2023-11-24T08:51:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.status 2023-11-24T08:51:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-11-24T08:51:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:51:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:51:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:51:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 119 = 2023-11-24T08:51:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:51:27.602 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7328815257279762) > weight (0.25) 2023-11-24T08:51:27.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.656179033669368) > weight (0.25) 2023-11-24T08:51:27.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9774450801066784) > weight (0.25) 2023-11-24T08:51:27.603 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-24T08:51:27.603 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-24T08:51:27.603 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-11-24T08:51:27.604 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T08:51:27.607+0000 7fe5e82d8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 51807) UID: 1000 2023-11-24T08:51:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.status 2023-11-24T08:51:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.status 2023-11-24T08:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-11-24T08:51:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=119 2023-11-24T08:51:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 119 = 119 2023-11-24T08:51:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-11-24T08:51:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:28.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:51:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-11-24T08:51:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_1': 2023-11-24T08:51:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:51:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:51:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:51:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10afc27fa0a1 2023-11-24T08:51:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10afc27fa0a1 2023-11-24T08:51:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:51:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:51:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:51:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:51:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:13 2023 2023-11-24T08:51:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:13 2023 2023-11-24T08:51:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:13 2023 2023-11-24T08:51:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:51:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:51:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:51:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 31318b39-6acb-4741-8297-a101c3fafabc 2023-11-24T08:51:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:51:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_1 snap 2023-11-24T08:51:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T08:51:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:51:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_1.export 2023-11-24T08:51:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.export 2023-11-24T08:51:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_1.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.export 2023-11-24T08:51:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_1.export 2023-11-24T08:51:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:51:29.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.export 2023-11-24T08:51:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:51:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_1.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.export 2023-11-24T08:51:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_1.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_1.export 2023-11-24T08:51:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2023-11-24T08:51:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_2 snap 2023-11-24T08:51:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:51:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:51:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ localgrep cluster=cluster1 -i 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:51:32.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:51:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:51:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:51:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:51:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:51:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-11-24T08:51:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:51:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:51:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:51:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:51:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:51:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:51:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-11-24T08:51:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:51:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:51:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-11-24T08:51:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_2.status 2023-11-24T08:51:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_2.status 2023-11-24T08:51:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_2.status 2023-11-24T08:51:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_2.status 2023-11-24T08:51:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_2.status 2023-11-24T08:51:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.status 2023-11-24T08:51:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-11-24T08:51:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-11-24T08:51:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 121 2023-11-24T08:51:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_2.status 2023-11-24T08:51:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_2.status 2023-11-24T08:51:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-11-24T08:51:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:33.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:51:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=121 2023-11-24T08:51:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:51:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.status 2023-11-24T08:51:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.status 2023-11-24T08:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-11-24T08:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=69 2023-11-24T08:51:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 121 = 69 2023-11-24T08:51:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:51:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.status 2023-11-24T08:51:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.status 2023-11-24T08:51:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-11-24T08:51:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:51:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:51:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:51:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 121 = 2023-11-24T08:51:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:51:34.604 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-24T08:51:34.605 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:51:34.605 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-24T08:51:34.605 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-24T08:51:34.605 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-24T08:51:34.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-24T08:51:34.608 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-11-24T08:51:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.status 2023-11-24T08:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.status 2023-11-24T08:51:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-11-24T08:51:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:51:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:51:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:51:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 121 = 2023-11-24T08:51:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:51:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.status 2023-11-24T08:51:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.status 2023-11-24T08:51:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-11-24T08:51:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:37.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=121 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 121 = 121 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:51:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-11-24T08:51:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_2': 2023-11-24T08:51:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:51:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:51:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:51:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c1ce103785 2023-11-24T08:51:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c1ce103785 2023-11-24T08:51:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:51:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:28 2023 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:28 2023 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:28 2023 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f38cb77f-9d83-40c5-a0c0-ff19cc94a15c 2023-11-24T08:51:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:51:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_2 snap 2023-11-24T08:51:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T08:51:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:51:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_2.export 2023-11-24T08:51:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.export 2023-11-24T08:51:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_2.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.export 2023-11-24T08:51:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_2.export 2023-11-24T08:51:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:51:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.export 2023-11-24T08:51:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:51:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_2.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.export 2023-11-24T08:51:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_2.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_2.export 2023-11-24T08:51:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2023-11-24T08:51:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_3 snap 2023-11-24T08:51:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:51:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-11-24T08:51:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-11-24T08:51:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:51:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:51:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-11-24T08:51:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:51:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:51:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + shift 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i+ statelocal 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: loop_instance 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:51:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:51:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-11-24T08:51:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:51:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:51:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:51:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-11-24T08:51:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:no valid command found; 2 closest matches: 2023-11-24T08:51:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_34 2023-11-24T08:51:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_38 2023-11-24T08:51:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:51:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:51:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:51:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:51:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-11-24T08:51:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:no valid command found; 3 closest matches: 2023-11-24T08:51:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_31 2023-11-24T08:51:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_32 2023-11-24T08:51:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_36 2023-11-24T08:51:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2023-11-24T08:51:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:51:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:51:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:51:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:51:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:51:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:51:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:51:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:51:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:51:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:51:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:51:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:51:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:51:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:51:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:51:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2023-11-24T08:51:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2023-11-24T08:51:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:51:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2023-11-24T08:51:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:51:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:51:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:51:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:42.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:51:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:51:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:51:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:51:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2023-11-24T08:51:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:51:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:51:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:51:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:51:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:51:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:51:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-11-24T08:51:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:51:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:51:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-11-24T08:51:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_3.status 2023-11-24T08:51:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_3.status 2023-11-24T08:51:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_3.status 2023-11-24T08:51:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_3.status 2023-11-24T08:51:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_3.status 2023-11-24T08:51:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.status 2023-11-24T08:51:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-11-24T08:51:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-11-24T08:51:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 123 2023-11-24T08:51:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_3.status 2023-11-24T08:51:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_3.status 2023-11-24T08:51:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-11-24T08:51:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:51:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=123 2023-11-24T08:51:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:51:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.status 2023-11-24T08:51:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.status 2023-11-24T08:51:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-11-24T08:51:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:51:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:51:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:51:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 123 = 2023-11-24T08:51:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:51:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.status 2023-11-24T08:51:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.status 2023-11-24T08:51:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-11-24T08:51:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:43.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:51:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:51:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:51:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 123 = 2023-11-24T08:51:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:51:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.status 2023-11-24T08:51:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.status 2023-11-24T08:51:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-11-24T08:51:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=123 2023-11-24T08:51:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 123 = 123 2023-11-24T08:51:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-11-24T08:51:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:51:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-11-24T08:51:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_3': 2023-11-24T08:51:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:51:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:51:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1207889205c 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1207889205c 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:39 2023 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:39 2023 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:39 2023 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:51:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:51:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 0910fa58-c0e2-4ebc-ba44-f7fe28530964 2023-11-24T08:51:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:51:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_3 snap 2023-11-24T08:51:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T08:51:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:51:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_3.export 2023-11-24T08:51:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.export 2023-11-24T08:51:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_3.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.export 2023-11-24T08:51:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_3.export 2023-11-24T08:51:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:51:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.export 2023-11-24T08:51:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:51:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_3.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.export 2023-11-24T08:51:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_3.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_3.export 2023-11-24T08:51:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2023-11-24T08:51:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_4 snap 2023-11-24T08:51:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:51:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-11-24T08:51:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-11-24T08:51:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:51:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:51:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-11-24T08:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:51:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:51:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:51:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-11-24T08:51:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ cluster_instance=cluster1 2023-11-24T08:51:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:51:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:51:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:51:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:51:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:51:48.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-11-24T08:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:51:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:51:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-11-24T08:51:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:no valid command found; 1 closest matches: 2023-11-24T08:51:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_42 2023-11-24T08:51:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:51:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:51:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:51:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-11-24T08:51:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:no valid command found; 3 closest matches: 2023-11-24T08:51:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_40 2023-11-24T08:51:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_44 2023-11-24T08:51:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_47 2023-11-24T08:51:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-11-24T08:51:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:51:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:51:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:51:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:51:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:51:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:51:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:51:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-11-24T08:51:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:no valid command found; 2 closest matches: 2023-11-24T08:51:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_41 2023-11-24T08:51:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_49 2023-11-24T08:51:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-11-24T08:51:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:51:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:51:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:51:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:51:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:51:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:51:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:51:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:51:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:51:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-11-24T08:51:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:no valid command found; 4 closest matches: 2023-11-24T08:51:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_43 2023-11-24T08:51:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_45 2023-11-24T08:51:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_46 2023-11-24T08:51:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_48 2023-11-24T08:51:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:51:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:51:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-11-24T08:51:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:51:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:51:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:51:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-11-24T08:51:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:51:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:51:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:51:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:51:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_4.status 2023-11-24T08:51:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_4.status 2023-11-24T08:51:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_4.status 2023-11-24T08:51:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_4.status 2023-11-24T08:51:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_4.status 2023-11-24T08:51:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.status 2023-11-24T08:51:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-11-24T08:51:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-11-24T08:51:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 125 2023-11-24T08:51:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_4.status 2023-11-24T08:51:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_4.status 2023-11-24T08:51:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-11-24T08:51:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:51:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=125 2023-11-24T08:51:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:51:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.status 2023-11-24T08:51:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.status 2023-11-24T08:51:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-11-24T08:51:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=71 2023-11-24T08:51:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 125 = 71 2023-11-24T08:51:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:51:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.status 2023-11-24T08:51:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.status 2023-11-24T08:51:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-11-24T08:51:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:51:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:51:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:51:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 125 = 2023-11-24T08:51:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:51:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.status 2023-11-24T08:51:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.status 2023-11-24T08:51:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-11-24T08:51:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:51:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:51:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:51:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 125 = 2023-11-24T08:51:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:51:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.status 2023-11-24T08:51:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.status 2023-11-24T08:51:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-11-24T08:51:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=125 2023-11-24T08:51:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 125 = 125 2023-11-24T08:51:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-11-24T08:51:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:51:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-11-24T08:51:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_4': 2023-11-24T08:51:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:51:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:51:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:51:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 126af4eb46e1 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.126af4eb46e1 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:46 2023 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:46 2023 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:46 2023 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:51:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: eb96c510-4f18-49d6-8bef-2744c8373fbf 2023-11-24T08:51:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:51:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_4 snap 2023-11-24T08:51:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T08:51:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:51:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_4.export 2023-11-24T08:51:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.export 2023-11-24T08:51:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_4.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.export 2023-11-24T08:51:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_4.export 2023-11-24T08:51:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:51:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.export 2023-11-24T08:51:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:51:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_4.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.export 2023-11-24T08:51:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_4.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_4.export 2023-11-24T08:51:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2023-11-24T08:51:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_5 snap 2023-11-24T08:51:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:51:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:51:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:51:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-11-24T08:51:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:51:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-11-24T08:51:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:51:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:51:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:51:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:51:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-11-24T08:51:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:51:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:51:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:51:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:51:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-11-24T08:51:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:51:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift+ 2023-11-24T08:51:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:51:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:51:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:51:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:51:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:51:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:58.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:51:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:51:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:51:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-11-24T08:51:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:no valid command found; 1 closest matches: 2023-11-24T08:51:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status mirror/image_50 2023-11-24T08:51:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-11-24T08:51:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:51:58.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:51:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:51:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:51:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:51:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-11-24T08:51:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-11-24T08:51:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:51:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:51:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:51:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:51:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:51:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:51:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:51:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:51:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-11-24T08:51:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-11-24T08:51:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:51:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:51:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2023-11-24T08:51:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:51:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:51:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:51:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:51:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:51:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:51:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:51:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:51:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:51:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:51:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:51:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:51:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:51:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:51:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:51:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:51:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:51:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:51:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:51:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2023-11-24T08:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:51:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:51:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:51:59.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:51:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:51:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_5.status 2023-11-24T08:51:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_5.status 2023-11-24T08:51:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_5.status 2023-11-24T08:51:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_5.status 2023-11-24T08:51:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_5.status 2023-11-24T08:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.status 2023-11-24T08:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-11-24T08:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:51:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-11-24T08:51:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 127 2023-11-24T08:51:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_5.status 2023-11-24T08:51:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:51:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:51:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_5.status 2023-11-24T08:51:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-11-24T08:51:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:51:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=127 2023-11-24T08:51:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:51:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.status 2023-11-24T08:51:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:51:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:51:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:51:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.status 2023-11-24T08:51:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-11-24T08:51:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:51:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:51:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:51:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:51:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:51:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:51:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:51:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 127 = 2023-11-24T08:51:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:52:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.status 2023-11-24T08:52:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:52:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.status 2023-11-24T08:52:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-11-24T08:52:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 127 = 2023-11-24T08:52:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:52:01.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.status 2023-11-24T08:52:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:52:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.status 2023-11-24T08:52:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-11-24T08:52:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 127 = 2023-11-24T08:52:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:52:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.status 2023-11-24T08:52:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:52:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.status 2023-11-24T08:52:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-11-24T08:52:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:02.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=127 2023-11-24T08:52:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 127 = 127 2023-11-24T08:52:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-11-24T08:52:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:52:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-11-24T08:52:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_5': 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10afa2ed54db 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10afa2ed54db 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:42:54 2023 2023-11-24T08:52:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:42:54 2023 2023-11-24T08:52:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:42:54 2023 2023-11-24T08:52:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:52:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:52:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:52:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 14adee4f-4c4d-4e0f-8207-88b9d7b5b7af 2023-11-24T08:52:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:52:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_5 snap 2023-11-24T08:52:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T08:52:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:52:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_5.export 2023-11-24T08:52:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.export 2023-11-24T08:52:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_5.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.export 2023-11-24T08:52:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_5.export 2023-11-24T08:52:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.export 2023-11-24T08:52:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_5.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.export 2023-11-24T08:52:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_5.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_5.export 2023-11-24T08:52:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2023-11-24T08:52:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_6 snap 2023-11-24T08:52:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:52:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-11-24T08:52:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-11-24T08:52:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:52:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-11-24T08:52:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:52:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:52:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:52:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-11-24T08:52:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:52:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:52:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ loop_instance 2023-11-24T08:52:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:52:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:52:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:52:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:52:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-11-24T08:52:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:52:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:52:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:52:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-11-24T08:52:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-11-24T08:52:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-11-24T08:52:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:52:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:52:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:52:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-11-24T08:52:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:52:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:52:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:52:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:52:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:52:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:52:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_6.status 2023-11-24T08:52:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_6.status 2023-11-24T08:52:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_6.status 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_6.status 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_6.status 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.status 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-11-24T08:52:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 129 2023-11-24T08:52:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_6.status 2023-11-24T08:52:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_6.status 2023-11-24T08:52:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-11-24T08:52:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:52:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=129 2023-11-24T08:52:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:52:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.status 2023-11-24T08:52:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.status 2023-11-24T08:52:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-11-24T08:52:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=73 2023-11-24T08:52:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 129 = 73 2023-11-24T08:52:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:52:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.status 2023-11-24T08:52:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.status 2023-11-24T08:52:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-11-24T08:52:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 129 = 2023-11-24T08:52:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:52:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.status 2023-11-24T08:52:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.status 2023-11-24T08:52:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-11-24T08:52:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 129 = 2023-11-24T08:52:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:52:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.status 2023-11-24T08:52:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.status 2023-11-24T08:52:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-11-24T08:52:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=129 2023-11-24T08:52:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 129 = 129 2023-11-24T08:52:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-11-24T08:52:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-11-24T08:52:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_6': 2023-11-24T08:52:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:52:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:52:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:52:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c161ef2a96 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c161ef2a96 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:09 2023 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:09 2023 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:09 2023 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f792a31d-dcf2-4828-bc5f-9490ec637a03 2023-11-24T08:52:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:52:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_6 snap 2023-11-24T08:52:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T08:52:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:52:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_6.export 2023-11-24T08:52:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.export 2023-11-24T08:52:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_6.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.export 2023-11-24T08:52:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_6.export 2023-11-24T08:52:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.export 2023-11-24T08:52:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_6.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.export 2023-11-24T08:52:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_6.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_6.export 2023-11-24T08:52:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2023-11-24T08:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_7 snap 2023-11-24T08:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:52:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-11-24T08:52:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-11-24T08:52:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:52:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-11-24T08:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:52:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:52:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:52:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-11-24T08:52:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ cluster_instance=cluster1 2023-11-24T08:52:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ localgrep cluster=cluster1 2023-11-24T08:52:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -i state 2023-11-24T08:52:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:52:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:52:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:52:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-11-24T08:52:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-11-24T08:52:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:16.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-11-24T08:52:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:52:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:52:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:52:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:52:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-11-24T08:52:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-11-24T08:52:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-11-24T08:52:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:52:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:52:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:52:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-11-24T08:52:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-11-24T08:52:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2023-11-24T08:52:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:52:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2023-11-24T08:52:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:52:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:52:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:52:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:52:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-11-24T08:52:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:52:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_7.status 2023-11-24T08:52:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_7.status 2023-11-24T08:52:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:52:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_7.status 2023-11-24T08:52:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_7.status 2023-11-24T08:52:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_7.status 2023-11-24T08:52:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:52:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.status 2023-11-24T08:52:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-11-24T08:52:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-11-24T08:52:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 131 2023-11-24T08:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_7.status 2023-11-24T08:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_7.status 2023-11-24T08:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-11-24T08:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:52:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=131 2023-11-24T08:52:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:52:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.status 2023-11-24T08:52:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.status 2023-11-24T08:52:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-11-24T08:52:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=74 2023-11-24T08:52:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 131 = 74 2023-11-24T08:52:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:52:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.status 2023-11-24T08:52:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.status 2023-11-24T08:52:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-11-24T08:52:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 131 = 2023-11-24T08:52:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:52:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.status 2023-11-24T08:52:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.status 2023-11-24T08:52:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-11-24T08:52:19.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 131 = 2023-11-24T08:52:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:52:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.status 2023-11-24T08:52:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.status 2023-11-24T08:52:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-11-24T08:52:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=131 2023-11-24T08:52:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 131 = 131 2023-11-24T08:52:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-11-24T08:52:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-11-24T08:52:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_7': 2023-11-24T08:52:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:52:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:52:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:52:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12072db2c805 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12072db2c805 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:18 2023 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:18 2023 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:18 2023 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:52:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:52:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:52:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: e07e8277-39ef-4a01-8ed4-6a47771afe3f 2023-11-24T08:52:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:52:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_7 snap 2023-11-24T08:52:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T08:52:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:52:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_7.export 2023-11-24T08:52:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.export 2023-11-24T08:52:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_7.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.export 2023-11-24T08:52:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_7.export 2023-11-24T08:52:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:22.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.export 2023-11-24T08:52:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_7.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.export 2023-11-24T08:52:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_7.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_7.export 2023-11-24T08:52:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2023-11-24T08:52:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_8 snap 2023-11-24T08:52:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:52:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-11-24T08:52:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-11-24T08:52:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:52:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-11-24T08:52:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:52:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:52:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:52:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_8 2023-11-24T08:52:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:52:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:52:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:52:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:52:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:52:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:52:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:52:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-11-24T08:52:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:52:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:52:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:52:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-11-24T08:52:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:52:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:52:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:52:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:52:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:52:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:52:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:52:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_8.status 2023-11-24T08:52:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_8.status 2023-11-24T08:52:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:52:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_8.status 2023-11-24T08:52:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_8.status 2023-11-24T08:52:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_8.status 2023-11-24T08:52:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:52:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.status 2023-11-24T08:52:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-11-24T08:52:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-11-24T08:52:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 133 2023-11-24T08:52:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_8.status 2023-11-24T08:52:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_8.status 2023-11-24T08:52:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-11-24T08:52:26.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:52:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=133 2023-11-24T08:52:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:52:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.status 2023-11-24T08:52:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.status 2023-11-24T08:52:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-11-24T08:52:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=75 2023-11-24T08:52:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 133 = 75 2023-11-24T08:52:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:52:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.status 2023-11-24T08:52:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.status 2023-11-24T08:52:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-11-24T08:52:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 133 = 2023-11-24T08:52:27.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:52:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.status 2023-11-24T08:52:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.status 2023-11-24T08:52:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-11-24T08:52:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 133 = 2023-11-24T08:52:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:52:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.status 2023-11-24T08:52:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.status 2023-11-24T08:52:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-11-24T08:52:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=133 2023-11-24T08:52:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 133 = 133 2023-11-24T08:52:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-11-24T08:52:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-11-24T08:52:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_8': 2023-11-24T08:52:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 126ab83b3790 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.126ab83b3790 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:32 2023 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:32 2023 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:32 2023 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:52:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:52:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:52:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 8b099305-4776-4fa1-8f3c-5e18bc48f849 2023-11-24T08:52:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:52:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_8 snap 2023-11-24T08:52:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T08:52:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:52:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_8.export 2023-11-24T08:52:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.export 2023-11-24T08:52:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_8.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.export 2023-11-24T08:52:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_8.export 2023-11-24T08:52:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.export 2023-11-24T08:52:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_8.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.export 2023-11-24T08:52:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_8.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_8.export 2023-11-24T08:52:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2023-11-24T08:52:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_9 snap 2023-11-24T08:52:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:52:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-11-24T08:52:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-11-24T08:52:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:52:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-11-24T08:52:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:52:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:52:34.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:52:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-11-24T08:52:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:52:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state+ 2023-11-24T08:52:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:local cluster=cluster1 2023-11-24T08:52:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:52:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:52:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:52:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:52:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:52:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-11-24T08:52:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:52:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:34.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:52:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-11-24T08:52:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:52:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:52:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:52:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:52:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:52:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:52:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-11-24T08:52:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:52:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:52:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:52:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-11-24T08:52:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:52:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_9.status 2023-11-24T08:52:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_9.status 2023-11-24T08:52:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:52:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_9.status 2023-11-24T08:52:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_9.status 2023-11-24T08:52:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_9.status 2023-11-24T08:52:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:52:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.status 2023-11-24T08:52:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-11-24T08:52:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-11-24T08:52:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 135 2023-11-24T08:52:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_9.status 2023-11-24T08:52:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_9.status 2023-11-24T08:52:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-11-24T08:52:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:52:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=135 2023-11-24T08:52:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:52:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.status 2023-11-24T08:52:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.status 2023-11-24T08:52:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-11-24T08:52:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 135 = 2023-11-24T08:52:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:52:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.status 2023-11-24T08:52:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.status 2023-11-24T08:52:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-11-24T08:52:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:36.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:52:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:52:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:52:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:52:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:52:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 135 = 2023-11-24T08:52:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:52:36.620 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4157984069982751) > weight (0.25) 2023-11-24T08:52:36.621 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-24T08:52:36.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-24T08:52:36.621 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-11-24T08:52:36.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T08:52:36.621+0000 7f0fe2393640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 51170) UID: 1000 2023-11-24T08:52:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.status 2023-11-24T08:52:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.status 2023-11-24T08:52:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-11-24T08:52:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:52:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:52:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=135 2023-11-24T08:52:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 135 = 135 2023-11-24T08:52:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-11-24T08:52:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-11-24T08:52:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_9': 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10afd79b5549 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10afd79b5549 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:52:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:39 2023 2023-11-24T08:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:39 2023 2023-11-24T08:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:39 2023 2023-11-24T08:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 18de629c-76f5-4bac-9f50-e3a0a137e23c 2023-11-24T08:52:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:52:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_9 snap 2023-11-24T08:52:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T08:52:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:52:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_9.export 2023-11-24T08:52:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.export 2023-11-24T08:52:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_9.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.export 2023-11-24T08:52:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_9.export 2023-11-24T08:52:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.export 2023-11-24T08:52:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:52:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_9.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.export 2023-11-24T08:52:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_9.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_9.export 2023-11-24T08:52:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:52:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2023-11-24T08:52:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_10 snap 2023-11-24T08:52:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:52:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:52:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:52:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:52:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-11-24T08:52:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:52:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:52:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:52:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:52:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:52:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror+ status mirror/image_10 2023-11-24T08:52:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:52:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:52:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:52:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:52:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:52:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:52:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:52:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-11-24T08:52:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:52:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:52:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:52:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-11-24T08:52:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-11-24T08:52:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:52:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:52:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:52:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-11-24T08:52:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-11-24T08:52:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:52:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-11-24T08:52:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2023-11-24T08:52:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:52:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2023-11-24T08:52:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:52:42.622 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-24T08:52:42.622 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:52:42.623 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-24T08:52:42.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-24T08:52:42.623 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-24T08:52:42.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-24T08:52:42.626 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2023-11-24T08:52:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:52:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:52:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-11-24T08:52:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:52:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:52:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:52:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-11-24T08:52:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:52:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:43.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-11-24T08:52:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-11-24T08:52:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:52:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:52:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-11-24T08:52:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2023-11-24T08:52:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:52:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:52:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:52:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:52:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:52:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:52:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2023-11-24T08:52:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:52:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:52:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:52:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-11-24T08:52:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:52:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:45.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:52:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-11-24T08:52:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-11-24T08:52:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:52:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:52:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:52:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-11-24T08:52:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-11-24T08:52:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:52:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:52:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:52:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:52:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:52:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:52:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-11-24T08:52:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2023-11-24T08:52:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:52:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:52:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:52:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:52:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:52:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2023-11-24T08:52:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T08:52:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:52:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:52:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:52:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-11-24T08:52:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-11-24T08:52:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:52:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:52:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:52:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:52:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-11-24T08:52:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-11-24T08:52:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:52:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:52:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:52:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:52:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-11-24T08:52:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2023-11-24T08:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:52:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2023-11-24T08:52:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:52:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:52:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:52:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:52:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:52:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:52:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-11-24T08:52:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-11-24T08:52:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:52:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:52:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:52:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:52:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:52:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-11-24T08:52:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-11-24T08:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:52:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:52:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:52:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:52:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:52:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:52:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-11-24T08:52:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:52:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:52:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:52:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:52:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:52:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:52:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:52:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:52:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:52:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2023-11-24T08:52:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:52:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:52:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:52:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:53:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:53:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:53:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-11-24T08:53:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:53:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-11-24T08:53:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:53:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:53:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-11-24T08:53:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:53:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2023-11-24T08:53:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:53:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:53:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:53:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:53:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2023-11-24T08:53:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:53:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:53:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:53:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:53:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:53:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:53:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:53:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:53:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:53:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2023-11-24T08:53:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:53:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:53:15.609 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4833989901891641) > weight (0.25) 2023-11-24T08:53:15.610 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9130918444479466) > weight (0.25) 2023-11-24T08:53:15.610 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5964400172903144) > weight (0.25) 2023-11-24T08:53:15.610 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-24T08:53:15.610 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-24T08:53:15.610 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-11-24T08:53:15.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T08:53:15.608+0000 7f80c3e4f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 53946) UID: 1000 2023-11-24T08:53:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:53:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:53:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-11-24T08:53:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:53:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:53:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:53:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:53:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:53:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:53:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-11-24T08:53:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-24T08:53:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-11-24T08:53:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:53:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:53:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:53:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:53:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:53:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:53:16.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-11-24T08:53:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:53:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:53:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:53:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:53:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:53:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_10.status 2023-11-24T08:53:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_10.status 2023-11-24T08:53:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:53:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_10.status 2023-11-24T08:53:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_10.status 2023-11-24T08:53:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_10.status 2023-11-24T08:53:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:53:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.status 2023-11-24T08:53:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-11-24T08:53:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-11-24T08:53:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 137 2023-11-24T08:53:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_10.status 2023-11-24T08:53:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_10.status 2023-11-24T08:53:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-11-24T08:53:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:53:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=137 2023-11-24T08:53:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:53:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.status 2023-11-24T08:53:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.status 2023-11-24T08:53:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-11-24T08:53:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=77 2023-11-24T08:53:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 137 = 77 2023-11-24T08:53:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:53:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.status 2023-11-24T08:53:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.status 2023-11-24T08:53:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-11-24T08:53:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=77 2023-11-24T08:53:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 137 = 77 2023-11-24T08:53:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:53:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.status 2023-11-24T08:53:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.status 2023-11-24T08:53:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-11-24T08:53:18.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:53:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:53:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:53:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 137 = 2023-11-24T08:53:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:53:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.status 2023-11-24T08:53:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.status 2023-11-24T08:53:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-11-24T08:53:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=137 2023-11-24T08:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 137 = 137 2023-11-24T08:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-11-24T08:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:20.611 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-24T08:53:20.611 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:53:20.611 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-24T08:53:20.611 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-24T08:53:20.612 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-24T08:53:20.614 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-24T08:53:20.615 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2023-11-24T08:53:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_10': 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c1fcbe14ae 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c1fcbe14ae 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:53:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:53:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:45 2023 2023-11-24T08:53:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:45 2023 2023-11-24T08:53:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:45 2023 2023-11-24T08:53:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:53:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:53:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:53:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 342ef9c2-57fd-4085-be72-fae4ddc7c8a0 2023-11-24T08:53:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:53:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_10 snap 2023-11-24T08:53:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T08:53:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:53:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_10.export 2023-11-24T08:53:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.export 2023-11-24T08:53:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_10.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.export 2023-11-24T08:53:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_10.export 2023-11-24T08:53:21.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.export 2023-11-24T08:53:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_10.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.export 2023-11-24T08:53:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_10.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_10.export 2023-11-24T08:53:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_11 2023-11-24T08:53:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_11 snap 2023-11-24T08:53:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:53:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-11-24T08:53:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:53:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:53:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:53:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-11-24T08:53:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:53:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state+ 2023-11-24T08:53:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:local cluster=cluster1 2023-11-24T08:53:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:53:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:53:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:53:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:53:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:53:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:53:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-11-24T08:53:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-11-24T08:53:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:53:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:53:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:53:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:53:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-11-24T08:53:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-11-24T08:53:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2023-11-24T08:53:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:53:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:53:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:24.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:53:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2023-11-24T08:53:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-11-24T08:53:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2023-11-24T08:53:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:53:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:53:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:53:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:53:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:53:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:53:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:53:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:53:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2023-11-24T08:53:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:53:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:53:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:53:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:53:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:53:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-11-24T08:53:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_11.status 2023-11-24T08:53:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_11.status 2023-11-24T08:53:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:53:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_11.status 2023-11-24T08:53:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_11.status 2023-11-24T08:53:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + echo cluster1-mirror-image_11.status 2023-11-24T08:53:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sed -e s|[/ ]|_|g 2023-11-24T08:53:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.status 2023-11-24T08:53:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-11-24T08:53:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-11-24T08:53:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 139 2023-11-24T08:53:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_11.status 2023-11-24T08:53:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_11.status 2023-11-24T08:53:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-11-24T08:53:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:53:25.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=139 2023-11-24T08:53:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:53:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.status 2023-11-24T08:53:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:25.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.status 2023-11-24T08:53:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-11-24T08:53:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:53:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:53:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:53:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 139 = 2023-11-24T08:53:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:53:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.status 2023-11-24T08:53:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.status 2023-11-24T08:53:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-11-24T08:53:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:53:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:53:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:53:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 139 = 2023-11-24T08:53:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:53:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.status 2023-11-24T08:53:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.status 2023-11-24T08:53:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-11-24T08:53:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=139 2023-11-24T08:53:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 139 = 139 2023-11-24T08:53:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-11-24T08:53:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-11-24T08:53:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_11': 2023-11-24T08:53:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:53:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:53:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:53:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120730e533b6 2023-11-24T08:53:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120730e533b6 2023-11-24T08:53:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:53:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:53:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:53:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:53:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:43:55 2023 2023-11-24T08:53:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:43:55 2023 2023-11-24T08:53:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:43:55 2023 2023-11-24T08:53:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:53:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:53:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:53:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 93be2a75-cd83-40a7-a3da-37d47064a00c 2023-11-24T08:53:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:53:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_11 snap 2023-11-24T08:53:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T08:53:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:53:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_11.export 2023-11-24T08:53:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.export 2023-11-24T08:53:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_11.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.export 2023-11-24T08:53:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_11.export 2023-11-24T08:53:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.export 2023-11-24T08:53:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_11.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.export 2023-11-24T08:53:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_11.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_11.export 2023-11-24T08:53:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_12 2023-11-24T08:53:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_12 snap 2023-11-24T08:53:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:53:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-11-24T08:53:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:53:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:53:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-11-24T08:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i+ local state instance=cluster1 2023-11-24T08:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:53:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:53:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:53:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:53:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:53:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:53:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:53:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:53:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-11-24T08:53:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-11-24T08:53:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-11-24T08:53:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:53:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:53:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:53:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:53:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:53:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:31.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:53:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-11-24T08:53:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-11-24T08:53:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-11-24T08:53:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:53:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:53:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-11-24T08:53:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:53:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:53:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:53:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:53:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:53:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-11-24T08:53:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_12.status 2023-11-24T08:53:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_12.status 2023-11-24T08:53:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:53:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_12.status 2023-11-24T08:53:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_12.status 2023-11-24T08:53:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_12.status 2023-11-24T08:53:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:53:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.status 2023-11-24T08:53:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-11-24T08:53:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-11-24T08:53:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 141 2023-11-24T08:53:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_12.status 2023-11-24T08:53:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_12.status 2023-11-24T08:53:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-11-24T08:53:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:53:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=141 2023-11-24T08:53:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:53:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.status 2023-11-24T08:53:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:32.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.status 2023-11-24T08:53:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-11-24T08:53:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=79 2023-11-24T08:53:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 141 = 79 2023-11-24T08:53:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:53:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.status 2023-11-24T08:53:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.status 2023-11-24T08:53:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-11-24T08:53:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:53:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:53:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:53:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 141 = 2023-11-24T08:53:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:53:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.status 2023-11-24T08:53:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.status 2023-11-24T08:53:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-11-24T08:53:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:53:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:53:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:53:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 141 = 2023-11-24T08:53:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:53:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.status 2023-11-24T08:53:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.status 2023-11-24T08:53:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-11-24T08:53:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=141 2023-11-24T08:53:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 141 = 141 2023-11-24T08:53:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-11-24T08:53:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-11-24T08:53:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_12': 2023-11-24T08:53:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:53:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:53:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:53:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 126a4df3b481 2023-11-24T08:53:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.126a4df3b481 2023-11-24T08:53:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:53:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:53:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:53:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:53:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:04 2023 2023-11-24T08:53:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:04 2023 2023-11-24T08:53:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:04 2023 2023-11-24T08:53:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:53:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:53:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:53:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 08429c44-38df-4fe3-933e-3a53c6f058ba 2023-11-24T08:53:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:53:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_12 snap 2023-11-24T08:53:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T08:53:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:53:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_12.export 2023-11-24T08:53:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.export 2023-11-24T08:53:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_12.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.export 2023-11-24T08:53:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_12.export 2023-11-24T08:53:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.export 2023-11-24T08:53:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_12.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.export 2023-11-24T08:53:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_12.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_12.export 2023-11-24T08:53:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_13 2023-11-24T08:53:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_13 snap 2023-11-24T08:53:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:53:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-11-24T08:53:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:53:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:53:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:53:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-11-24T08:53:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:53:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:53:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:53:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:53:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i state 2023-11-24T08:53:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:53:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:53:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-11-24T08:53:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:53:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:53:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:53:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-11-24T08:53:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-11-24T08:53:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-11-24T08:53:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:53:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:53:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-11-24T08:53:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-11-24T08:53:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2023-11-24T08:53:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:53:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:53:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:53:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:53:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:53:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:53:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2023-11-24T08:53:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:53:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + grep -i Replaying 2023-11-24T08:53:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo "state": "Replaying" 2023-11-24T08:53:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-11-24T08:53:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_13.status 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sed -e s|[/ ]|_|g 2023-11-24T08:53:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo cluster2-mirror-image_13.status 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_13.status 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_13.status 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_13.status 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.status 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-11-24T08:53:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 143 2023-11-24T08:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_13.status 2023-11-24T08:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_13.status 2023-11-24T08:53:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-11-24T08:53:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:53:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=143 2023-11-24T08:53:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:53:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.status 2023-11-24T08:53:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.status 2023-11-24T08:53:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-11-24T08:53:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=80 2023-11-24T08:53:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 143 = 80 2023-11-24T08:53:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:53:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.status 2023-11-24T08:53:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.status 2023-11-24T08:53:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-11-24T08:53:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:53:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:53:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:53:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 143 = 2023-11-24T08:53:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:53:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.status 2023-11-24T08:53:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.status 2023-11-24T08:53:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-11-24T08:53:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:53:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:53:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:53:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 143 = 2023-11-24T08:53:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:53:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.status 2023-11-24T08:53:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.status 2023-11-24T08:53:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-11-24T08:53:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=143 2023-11-24T08:53:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 143 = 143 2023-11-24T08:53:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-11-24T08:53:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-11-24T08:53:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_13': 2023-11-24T08:53:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:53:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:53:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:53:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10af833f8d32 2023-11-24T08:53:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10af833f8d32 2023-11-24T08:53:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:53:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:53:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:53:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:53:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:10 2023 2023-11-24T08:53:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:10 2023 2023-11-24T08:53:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:10 2023 2023-11-24T08:53:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:53:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:53:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:53:47.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 25e23892-6380-4923-94d3-6e58c6e982f0 2023-11-24T08:53:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:53:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_13 snap 2023-11-24T08:53:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T08:53:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:53:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_13.export 2023-11-24T08:53:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.export 2023-11-24T08:53:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_13.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.export 2023-11-24T08:53:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_13.export 2023-11-24T08:53:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.export 2023-11-24T08:53:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_13.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.export 2023-11-24T08:53:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_13.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_13.export 2023-11-24T08:53:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_14 2023-11-24T08:53:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_14 snap 2023-11-24T08:53:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:53:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-11-24T08:53:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-11-24T08:53:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:53:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-11-24T08:53:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-11-24T08:53:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:53:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:53:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:53:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:53:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:53:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:53:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:53:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-11-24T08:53:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:53:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:53:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:53:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-11-24T08:53:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-11-24T08:53:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:53:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:53:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-11-24T08:53:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:53:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:53:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:53:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:53:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:53:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:53:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:53:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:53:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-11-24T08:53:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:53:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:53:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:53:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:53:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-11-24T08:53:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_14.status 2023-11-24T08:53:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_14.status 2023-11-24T08:53:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:53:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_14.status 2023-11-24T08:53:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_14.status 2023-11-24T08:53:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_14.status 2023-11-24T08:53:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:53:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.status 2023-11-24T08:53:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-11-24T08:53:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-11-24T08:53:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 145 2023-11-24T08:53:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_14.status 2023-11-24T08:53:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_14.status 2023-11-24T08:53:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-11-24T08:53:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:53:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=145 2023-11-24T08:53:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:53:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.status 2023-11-24T08:53:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.status 2023-11-24T08:53:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-11-24T08:53:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=81 2023-11-24T08:53:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 145 = 81 2023-11-24T08:53:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:53:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.status 2023-11-24T08:53:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.status 2023-11-24T08:53:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-11-24T08:53:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:53:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:53:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:53:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 145 = 2023-11-24T08:53:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:53:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.status 2023-11-24T08:53:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.status 2023-11-24T08:53:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-11-24T08:53:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:53:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:53:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:53:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:53:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 145 = 2023-11-24T08:53:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:53:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.status 2023-11-24T08:53:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.status 2023-11-24T08:53:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-11-24T08:53:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=145 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 145 = 145 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-11-24T08:53:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_14': 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c118dee601 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c118dee601 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:19 2023 2023-11-24T08:53:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:19 2023 2023-11-24T08:53:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:19 2023 2023-11-24T08:53:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:53:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:53:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:53:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 2cfb09ab-c214-4e47-89a7-d051e9682117 2023-11-24T08:53:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:53:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_14 snap 2023-11-24T08:53:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T08:53:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:53:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_14.export 2023-11-24T08:53:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.export 2023-11-24T08:53:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_14.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.export 2023-11-24T08:53:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_14.export 2023-11-24T08:53:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.export 2023-11-24T08:53:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:53:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_14.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.export 2023-11-24T08:53:55.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_14.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_14.export 2023-11-24T08:53:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_15 2023-11-24T08:53:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_15 snap 2023-11-24T08:53:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:53:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:53:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:53:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-11-24T08:53:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:53:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-11-24T08:53:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:53:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:53:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:53:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:53:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-11-24T08:53:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:53:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:53:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:53:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:53:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-11-24T08:53:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:53:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:53:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:53:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:53:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:53:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:53:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:53:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:53:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:53:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:53:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:53:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:53:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:53:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:53:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:53:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-11-24T08:53:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:53:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:53:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:53:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:53:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:53:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:53:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:53:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-11-24T08:53:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_15.status 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sed -e s|[/ ]|_|g 2023-11-24T08:53:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo cluster2-mirror-image_15.status 2023-11-24T08:53:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_15.status 2023-11-24T08:53:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_15.status 2023-11-24T08:53:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_15.status 2023-11-24T08:53:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:53:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.status 2023-11-24T08:53:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-11-24T08:53:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:53:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-11-24T08:53:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 147 2023-11-24T08:53:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_15.status 2023-11-24T08:53:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:53:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:53:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_15.status 2023-11-24T08:53:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-11-24T08:53:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:53:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=147 2023-11-24T08:53:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:53:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:53:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.status 2023-11-24T08:53:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:53:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:53:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:53:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.status 2023-11-24T08:53:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-11-24T08:53:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:53:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:53:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=82 2023-11-24T08:53:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 147 = 82 2023-11-24T08:53:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:54:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.status 2023-11-24T08:54:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:54:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.status 2023-11-24T08:54:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-11-24T08:54:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 147 = 2023-11-24T08:54:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:54:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.status 2023-11-24T08:54:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:54:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.status 2023-11-24T08:54:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-11-24T08:54:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 147 = 2023-11-24T08:54:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:54:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.status 2023-11-24T08:54:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:54:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.status 2023-11-24T08:54:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-11-24T08:54:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=147 2023-11-24T08:54:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 147 = 147 2023-11-24T08:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-11-24T08:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_15': 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1207b51da6dd 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1207b51da6dd 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:54:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:54:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:25 2023 2023-11-24T08:54:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:25 2023 2023-11-24T08:54:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:25 2023 2023-11-24T08:54:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:54:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:54:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:54:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: f1cebaf0-aa03-4dea-9666-f49bdf41a7b2 2023-11-24T08:54:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:54:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_15 snap 2023-11-24T08:54:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T08:54:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:54:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_15.export 2023-11-24T08:54:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.export 2023-11-24T08:54:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_15.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.export 2023-11-24T08:54:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_15.export 2023-11-24T08:54:04.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.export 2023-11-24T08:54:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_15.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.export 2023-11-24T08:54:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_15.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_15.export 2023-11-24T08:54:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_16 2023-11-24T08:54:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_16 snap 2023-11-24T08:54:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:54:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-11-24T08:54:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-11-24T08:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-11-24T08:54:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:54:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:54:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:54:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-11-24T08:54:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:54:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grepshift 2023-11-24T08:54:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -i state 2023-11-24T08:54:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:54:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:54:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:54:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:54:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:54:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:54:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-11-24T08:54:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-11-24T08:54:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2023-11-24T08:54:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:54:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:54:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:54:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:54:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2023-11-24T08:54:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-11-24T08:54:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:54:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:54:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:54:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2023-11-24T08:54:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:54:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:54:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:54:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:54:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:54:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_16.status 2023-11-24T08:54:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + echo cluster2-mirror-image_16.statussed 2023-11-24T08:54:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -e s|[/ ]|_|g 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_16.status 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_16.status 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_16.status 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.status 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-11-24T08:54:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 149 2023-11-24T08:54:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_16.status 2023-11-24T08:54:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_16.status 2023-11-24T08:54:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-11-24T08:54:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:54:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=149 2023-11-24T08:54:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:54:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.status 2023-11-24T08:54:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.status 2023-11-24T08:54:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-11-24T08:54:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=83 2023-11-24T08:54:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 149 = 83 2023-11-24T08:54:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:54:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.status 2023-11-24T08:54:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.status 2023-11-24T08:54:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-11-24T08:54:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 149 = 2023-11-24T08:54:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:54:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.status 2023-11-24T08:54:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.status 2023-11-24T08:54:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-11-24T08:54:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 149 = 2023-11-24T08:54:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:54:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.status 2023-11-24T08:54:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.status 2023-11-24T08:54:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-11-24T08:54:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=149 2023-11-24T08:54:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 149 = 149 2023-11-24T08:54:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-11-24T08:54:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_16': 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 126a986b6c1b 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.126a986b6c1b 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:36 2023 2023-11-24T08:54:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:36 2023 2023-11-24T08:54:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:36 2023 2023-11-24T08:54:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:54:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:54:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:54:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 00787dee-37ed-4352-8685-7ea4d59a610b 2023-11-24T08:54:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:54:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_16 snap 2023-11-24T08:54:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T08:54:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:54:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_16.export 2023-11-24T08:54:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.export 2023-11-24T08:54:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_16.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.export 2023-11-24T08:54:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_16.export 2023-11-24T08:54:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.export 2023-11-24T08:54:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_16.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.export 2023-11-24T08:54:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_16.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_16.export 2023-11-24T08:54:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_17 2023-11-24T08:54:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_17 snap 2023-11-24T08:54:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:54:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-11-24T08:54:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-11-24T08:54:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:54:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-11-24T08:54:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:54:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:54:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sleep 0 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:54:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-11-24T08:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:54:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:54:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-11-24T08:54:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-11-24T08:54:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-11-24T08:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:54:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:54:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:54:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:54:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-11-24T08:54:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-11-24T08:54:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2023-11-24T08:54:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:54:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:54:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:54:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:54:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2023-11-24T08:54:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-11-24T08:54:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:54:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:54:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:54:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:54:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:54:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:54:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:54:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:54:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2023-11-24T08:54:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:54:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:54:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:54:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:54:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:54:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:54:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-11-24T08:54:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_17.status 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_17.status 2023-11-24T08:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_17.status 2023-11-24T08:54:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_17.status 2023-11-24T08:54:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + echo cluster1-mirror-image_17.status 2023-11-24T08:54:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sed -e s|[/ ]|_|g 2023-11-24T08:54:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.status 2023-11-24T08:54:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-11-24T08:54:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-11-24T08:54:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 151 2023-11-24T08:54:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_17.status 2023-11-24T08:54:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_17.status 2023-11-24T08:54:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-11-24T08:54:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:54:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=151 2023-11-24T08:54:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:54:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.status 2023-11-24T08:54:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.status 2023-11-24T08:54:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-11-24T08:54:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=84 2023-11-24T08:54:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 151 = 84 2023-11-24T08:54:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:54:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.status 2023-11-24T08:54:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.status 2023-11-24T08:54:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-11-24T08:54:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 151 = 2023-11-24T08:54:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:54:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.status 2023-11-24T08:54:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.status 2023-11-24T08:54:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-11-24T08:54:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 151 = 2023-11-24T08:54:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:54:20.627 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7842620842615722) > weight (0.25) 2023-11-24T08:54:20.628 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.953263215536025) > weight (0.25) 2023-11-24T08:54:20.628 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8192807371256475) > weight (0.25) 2023-11-24T08:54:20.628 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.2813820527988298) > weight (0.25) 2023-11-24T08:54:20.628 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-11-24T08:54:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.status 2023-11-24T08:54:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.status 2023-11-24T08:54:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-11-24T08:54:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=151 2023-11-24T08:54:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 151 = 151 2023-11-24T08:54:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-11-24T08:54:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-11-24T08:54:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_17': 2023-11-24T08:54:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:54:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:54:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:54:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10af9eefc7ed 2023-11-24T08:54:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10af9eefc7ed 2023-11-24T08:54:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:54:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:54:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:54:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:54:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:45 2023 2023-11-24T08:54:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:45 2023 2023-11-24T08:54:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:45 2023 2023-11-24T08:54:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:54:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:54:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:54:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 12e91ddf-7a62-43f6-8172-24011e6b5d43 2023-11-24T08:54:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:54:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_17 snap 2023-11-24T08:54:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T08:54:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:54:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_17.export 2023-11-24T08:54:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.export 2023-11-24T08:54:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_17.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.export 2023-11-24T08:54:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_17.export 2023-11-24T08:54:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.export 2023-11-24T08:54:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_17.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.export 2023-11-24T08:54:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_17.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_17.export 2023-11-24T08:54:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_18 2023-11-24T08:54:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_18 snap 2023-11-24T08:54:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:54:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-11-24T08:54:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:54:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:54:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror+ status mirror/image_18 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i state 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:local cluster_instance=cluster1 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:54:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:25.641 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:54:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-11-24T08:54:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:54:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:54:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:54:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:54:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:54:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:54:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-11-24T08:54:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:54:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:54:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-11-24T08:54:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_18.status 2023-11-24T08:54:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_18.status 2023-11-24T08:54:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_18.status 2023-11-24T08:54:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_18.status 2023-11-24T08:54:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_18.status 2023-11-24T08:54:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.status 2023-11-24T08:54:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-11-24T08:54:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-11-24T08:54:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 153 2023-11-24T08:54:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_18.status 2023-11-24T08:54:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_18.status 2023-11-24T08:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-11-24T08:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:54:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=153 2023-11-24T08:54:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:54:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.status 2023-11-24T08:54:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.status 2023-11-24T08:54:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-11-24T08:54:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=85 2023-11-24T08:54:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 153 = 85 2023-11-24T08:54:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:54:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.status 2023-11-24T08:54:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.status 2023-11-24T08:54:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-11-24T08:54:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=85 2023-11-24T08:54:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 153 = 85 2023-11-24T08:54:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:54:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.status 2023-11-24T08:54:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.status 2023-11-24T08:54:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-11-24T08:54:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 153 = 2023-11-24T08:54:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:54:28.360 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:54:28.355+0000 7f42b0626640 -1 reset not still connected to 0x561a70afba00 2023-11-24T08:54:28.360 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:54:28.355+0000 7f42b0626640 -1 reset not still connected to 0x561a73aeb930 2023-11-24T08:54:28.360 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:54:28.355+0000 7f42b0626640 -1 reset not still connected to 0x561a780da000 2023-11-24T08:54:28.367 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:54:28.363+0000 7f42b0626640 -1 reset not still connected to 0x561a780dbc70 2023-11-24T08:54:28.375 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:54:28.367+0000 7f42b0626640 -1 reset not still connected to 0x561a7a9000d0 2023-11-24T08:54:28.375 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:54:28.367+0000 7f42b0626640 -1 reset not still connected to 0x561a7a900340 2023-11-24T08:54:28.375 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:54:28.367+0000 7f42b0626640 -1 reset not still connected to 0x561a8af78c30 2023-11-24T08:54:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.status 2023-11-24T08:54:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.status 2023-11-24T08:54:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-11-24T08:54:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=153 2023-11-24T08:54:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 153 = 153 2023-11-24T08:54:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-11-24T08:54:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:30.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-11-24T08:54:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_18': 2023-11-24T08:54:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:54:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:54:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:54:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c1a5ac45fb 2023-11-24T08:54:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c1a5ac45fb 2023-11-24T08:54:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:54:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:44:56 2023 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:44:56 2023 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:44:56 2023 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: de67f348-91f1-4254-a508-55ce6ec8848d 2023-11-24T08:54:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:54:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_18 snap 2023-11-24T08:54:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T08:54:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:54:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_18.export 2023-11-24T08:54:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.export 2023-11-24T08:54:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_18.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.export 2023-11-24T08:54:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_18.export 2023-11-24T08:54:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.export 2023-11-24T08:54:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_18.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.export 2023-11-24T08:54:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_18.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_18.export 2023-11-24T08:54:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_19 2023-11-24T08:54:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_19 snap 2023-11-24T08:54:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:54:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-11-24T08:54:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:54:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:54:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:54:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + admin_daemons cluster1grep rbd -i state mirror 2023-11-24T08:54:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: status mirror/image_19 2023-11-24T08:54:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:54:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:54:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:54:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:54:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:54:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:54:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:34.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:54:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:54:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:54:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:54:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-11-24T08:54:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-11-24T08:54:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:34.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:54:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-11-24T08:54:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-11-24T08:54:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:54:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:54:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:54:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2023-11-24T08:54:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:54:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + echo "state": "Replaying" 2023-11-24T08:54:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i Replaying 2023-11-24T08:54:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:54:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:54:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_19.status 2023-11-24T08:54:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_19.status 2023-11-24T08:54:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_19.status 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_19.status 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_19.status 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.status 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-11-24T08:54:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 155 2023-11-24T08:54:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_19.status 2023-11-24T08:54:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_19.status 2023-11-24T08:54:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-11-24T08:54:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:54:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=155 2023-11-24T08:54:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:54:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.status 2023-11-24T08:54:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.status 2023-11-24T08:54:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-11-24T08:54:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 155 = 2023-11-24T08:54:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:54:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.status 2023-11-24T08:54:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.status 2023-11-24T08:54:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-11-24T08:54:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:36.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 155 = 2023-11-24T08:54:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:54:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.status 2023-11-24T08:54:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.status 2023-11-24T08:54:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-11-24T08:54:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=155 2023-11-24T08:54:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 155 = 155 2023-11-24T08:54:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-11-24T08:54:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-11-24T08:54:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_19': 2023-11-24T08:54:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:54:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1287ea4cab49 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1287ea4cab49 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:10 2023 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:10 2023 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:10 2023 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:54:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:54:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 7bc813da-10f2-42ef-b0da-2017c7edc945 2023-11-24T08:54:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:54:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_19 snap 2023-11-24T08:54:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T08:54:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:54:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_19.export 2023-11-24T08:54:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.export 2023-11-24T08:54:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_19.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.export 2023-11-24T08:54:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_19.export 2023-11-24T08:54:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.export 2023-11-24T08:54:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_19.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.export 2023-11-24T08:54:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_19.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_19.export 2023-11-24T08:54:38.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_20 2023-11-24T08:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_20 snap 2023-11-24T08:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:54:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-11-24T08:54:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-11-24T08:54:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:54:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-11-24T08:54:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:54:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:54:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:54:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-11-24T08:54:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:54:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + shift 2023-11-24T08:54:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ localgrep instance=cluster1 -i 2023-11-24T08:54:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state+ 2023-11-24T08:54:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:local loop_instance 2023-11-24T08:54:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:54:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:54:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:54:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-11-24T08:54:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:54:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:54:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:54:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:54:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:54:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:54:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_20.status 2023-11-24T08:54:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_20.status 2023-11-24T08:54:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_20.status 2023-11-24T08:54:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_20.status 2023-11-24T08:54:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_20.status 2023-11-24T08:54:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.status 2023-11-24T08:54:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-11-24T08:54:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-11-24T08:54:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 157 2023-11-24T08:54:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_20.status 2023-11-24T08:54:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_20.status 2023-11-24T08:54:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-11-24T08:54:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:54:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=157 2023-11-24T08:54:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:54:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.status 2023-11-24T08:54:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.status 2023-11-24T08:54:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-11-24T08:54:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=87 2023-11-24T08:54:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 157 = 87 2023-11-24T08:54:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:54:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.status 2023-11-24T08:54:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.status 2023-11-24T08:54:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-11-24T08:54:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 157 = 2023-11-24T08:54:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:54:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.status 2023-11-24T08:54:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.status 2023-11-24T08:54:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-11-24T08:54:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 157 = 2023-11-24T08:54:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:54:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.status 2023-11-24T08:54:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.status 2023-11-24T08:54:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-11-24T08:54:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=157 2023-11-24T08:54:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 157 = 157 2023-11-24T08:54:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-11-24T08:54:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-11-24T08:54:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_20': 2023-11-24T08:54:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:54:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:54:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:54:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12875f2bb36f 2023-11-24T08:54:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12875f2bb36f 2023-11-24T08:54:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:54:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:54:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:54:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:54:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:16 2023 2023-11-24T08:54:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:16 2023 2023-11-24T08:54:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:16 2023 2023-11-24T08:54:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:54:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:54:47.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:54:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9f9af0b3-95c5-474e-831d-a21ecd5dc147 2023-11-24T08:54:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:54:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_20 snap 2023-11-24T08:54:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T08:54:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:54:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_20.export 2023-11-24T08:54:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.export 2023-11-24T08:54:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_20.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.export 2023-11-24T08:54:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_20.export 2023-11-24T08:54:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.export 2023-11-24T08:54:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_20.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.export 2023-11-24T08:54:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_20.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_20.export 2023-11-24T08:54:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_21 2023-11-24T08:54:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_21 snap 2023-11-24T08:54:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:54:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-11-24T08:54:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-11-24T08:54:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:54:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-11-24T08:54:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:54:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:54:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:54:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-11-24T08:54:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:54:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i+ statelocal 2023-11-24T08:54:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: cluster=cluster1 2023-11-24T08:54:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:54:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:54:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:54:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:54:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:54:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-11-24T08:54:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:54:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:54:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:54:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-11-24T08:54:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-11-24T08:54:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-11-24T08:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:54:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:54:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-11-24T08:54:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-11-24T08:54:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2023-11-24T08:54:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:54:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:54:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:54:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:54:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:54:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:54:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:54:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:54:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:54:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:54:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:54:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:54:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:54:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:54:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2023-11-24T08:54:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:54:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:54:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_21.status 2023-11-24T08:54:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_21.status 2023-11-24T08:54:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_21.status 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_21.status 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_21.status 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.status 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-11-24T08:54:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 159 2023-11-24T08:54:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_21.status 2023-11-24T08:54:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_21.status 2023-11-24T08:54:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-11-24T08:54:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:54:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=159 2023-11-24T08:54:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:54:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.status 2023-11-24T08:54:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.status 2023-11-24T08:54:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-11-24T08:54:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=88 2023-11-24T08:54:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 159 = 88 2023-11-24T08:54:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.status 2023-11-24T08:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.status 2023-11-24T08:54:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-11-24T08:54:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=88 2023-11-24T08:54:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 159 = 88 2023-11-24T08:54:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.status 2023-11-24T08:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.status 2023-11-24T08:54:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-11-24T08:54:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:54:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:54:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:54:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:54:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 159 = 2023-11-24T08:54:53.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:54:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.status 2023-11-24T08:54:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.status 2023-11-24T08:54:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-11-24T08:54:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:54:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:54:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=159 2023-11-24T08:54:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 159 = 159 2023-11-24T08:54:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-11-24T08:54:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-11-24T08:54:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_21': 2023-11-24T08:54:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 10af4afcdfe0 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.10af4afcdfe0 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:31 2023 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:31 2023 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:31 2023 2023-11-24T08:54:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:54:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:54:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:54:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 3cc6e0f9-790b-43bf-a15d-2b4b68e157bd 2023-11-24T08:54:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:54:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_21 snap 2023-11-24T08:54:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T08:54:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:54:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_21.export 2023-11-24T08:54:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.export 2023-11-24T08:54:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_21.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.export 2023-11-24T08:54:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_21.export 2023-11-24T08:54:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.export 2023-11-24T08:54:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:54:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_21.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.export 2023-11-24T08:54:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_21.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_21.export 2023-11-24T08:54:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:54:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_22 2023-11-24T08:54:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_22 snap 2023-11-24T08:54:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:54:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:54:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:54:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:54:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:54:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-11-24T08:54:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:54:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:54:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:54:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:54:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + shift 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep+ -ilocal state loop_instance 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:54:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:54:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-11-24T08:54:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:54:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:54:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:54:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:54:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:54:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-11-24T08:54:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-11-24T08:54:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:54:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:54:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-11-24T08:54:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:54:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:54:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:54:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:54:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:54:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:54:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:54:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:54:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:54:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:54:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:54:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:54:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:54:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:54:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:54:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:54:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-11-24T08:55:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:55:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + grep -i Replaying 2023-11-24T08:55:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo "state": "Replaying" 2023-11-24T08:55:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-11-24T08:55:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_22.status 2023-11-24T08:55:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_22.status 2023-11-24T08:55:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:55:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_22.status 2023-11-24T08:55:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_22.status 2023-11-24T08:55:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_22.status 2023-11-24T08:55:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:55:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.status 2023-11-24T08:55:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-11-24T08:55:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-11-24T08:55:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 161 2023-11-24T08:55:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_22.status 2023-11-24T08:55:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_22.status 2023-11-24T08:55:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-11-24T08:55:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:55:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=161 2023-11-24T08:55:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:55:00.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.status 2023-11-24T08:55:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.status 2023-11-24T08:55:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-11-24T08:55:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=89 2023-11-24T08:55:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 161 = 89 2023-11-24T08:55:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:55:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.status 2023-11-24T08:55:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.status 2023-11-24T08:55:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-11-24T08:55:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=89 2023-11-24T08:55:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 161 = 89 2023-11-24T08:55:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:55:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.status 2023-11-24T08:55:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.status 2023-11-24T08:55:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-11-24T08:55:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:55:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:55:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:55:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:55:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 161 = 2023-11-24T08:55:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:55:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.status 2023-11-24T08:55:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.status 2023-11-24T08:55:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-11-24T08:55:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=161 2023-11-24T08:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 161 = 161 2023-11-24T08:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-11-24T08:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-11-24T08:55:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_22': 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c16a57a51f 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c16a57a51f 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:45 2023 2023-11-24T08:55:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:45 2023 2023-11-24T08:55:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:45 2023 2023-11-24T08:55:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:55:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:55:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:55:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 7c3b3b3c-f77d-402e-adec-8115e36a2f88 2023-11-24T08:55:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:55:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_22 snap 2023-11-24T08:55:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T08:55:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:55:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_22.export 2023-11-24T08:55:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.export 2023-11-24T08:55:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_22.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.export 2023-11-24T08:55:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_22.export 2023-11-24T08:55:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.export 2023-11-24T08:55:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_22.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.export 2023-11-24T08:55:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_22.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_22.export 2023-11-24T08:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_23 2023-11-24T08:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_23 snap 2023-11-24T08:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:55:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-11-24T08:55:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-11-24T08:55:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:55:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-11-24T08:55:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:55:08.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:55:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:55:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-11-24T08:55:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:55:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ cluster=cluster1 2023-11-24T08:55:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:55:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:55:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep+ -i state 2023-11-24T08:55:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sleep 0 2023-11-24T08:55:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:55:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:55:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-11-24T08:55:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:55:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:55:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:55:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:55:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:08.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:55:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-11-24T08:55:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-11-24T08:55:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-11-24T08:55:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:55:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:55:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:55:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:55:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-11-24T08:55:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-11-24T08:55:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:55:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2023-11-24T08:55:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:55:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:55:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:55:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:55:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-11-24T08:55:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_23.status 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_23.status 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_23.status 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_23.status 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + echo cluster1-mirror-image_23.status 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sed -e s|[/ ]|_|g 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.status 2023-11-24T08:55:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-11-24T08:55:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-11-24T08:55:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 163 2023-11-24T08:55:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_23.status 2023-11-24T08:55:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_23.status 2023-11-24T08:55:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-11-24T08:55:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:55:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=163 2023-11-24T08:55:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:55:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.status 2023-11-24T08:55:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.status 2023-11-24T08:55:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-11-24T08:55:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=90 2023-11-24T08:55:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 163 = 90 2023-11-24T08:55:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:55:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.status 2023-11-24T08:55:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.status 2023-11-24T08:55:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-11-24T08:55:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=90 2023-11-24T08:55:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 163 = 90 2023-11-24T08:55:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:55:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.status 2023-11-24T08:55:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.status 2023-11-24T08:55:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-11-24T08:55:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:55:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:55:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:55:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:55:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 163 = 2023-11-24T08:55:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:55:12.875 INFO:tasks.ceph.cluster1.osd.1.smithi153.stderr:2023-11-24T08:55:12.874+0000 7f9978959640 -1 reset not still connected to 0x55deff813d40 2023-11-24T08:55:12.875 INFO:tasks.ceph.cluster1.osd.1.smithi153.stderr:2023-11-24T08:55:12.874+0000 7f9978959640 -1 reset not still connected to 0x55df03767e10 2023-11-24T08:55:12.875 INFO:tasks.ceph.cluster1.osd.1.smithi153.stderr:2023-11-24T08:55:12.874+0000 7f9978959640 -1 reset not still connected to 0x55df05011040 2023-11-24T08:55:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.status 2023-11-24T08:55:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.status 2023-11-24T08:55:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-11-24T08:55:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=163 2023-11-24T08:55:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 163 = 163 2023-11-24T08:55:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-11-24T08:55:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:13.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-11-24T08:55:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_23': 2023-11-24T08:55:14.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12072331df46 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12072331df46 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:45:57 2023 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:45:57 2023 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:45:57 2023 2023-11-24T08:55:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:55:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:55:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:55:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 6e18b899-53e6-40ab-b047-1c57f3a06280 2023-11-24T08:55:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:55:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_23 snap 2023-11-24T08:55:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T08:55:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:55:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_23.export 2023-11-24T08:55:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.export 2023-11-24T08:55:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_23.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.export 2023-11-24T08:55:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_23.export 2023-11-24T08:55:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.export 2023-11-24T08:55:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_23.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.export 2023-11-24T08:55:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_23.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_23.export 2023-11-24T08:55:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_24 2023-11-24T08:55:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_24 snap 2023-11-24T08:55:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:55:16.616 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.45480838280677205) > weight (0.25) 2023-11-24T08:55:16.616 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-24T08:55:16.616 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-24T08:55:16.616 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-11-24T08:55:16.617 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-24T08:55:16.617 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:55:16.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T08:55:16.618+0000 7f7ff4699640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 53038) UID: 1000 2023-11-24T08:55:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-11-24T08:55:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-11-24T08:55:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:55:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-11-24T08:55:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:55:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:55:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:55:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-11-24T08:55:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:55:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:55:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:55:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sleep 0 2023-11-24T08:55:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:55:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:55:17.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:55:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-11-24T08:55:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:55:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-11-24T08:55:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-11-24T08:55:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:55:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-11-24T08:55:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-11-24T08:55:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:55:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-11-24T08:55:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-11-24T08:55:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_24 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:55:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_24 2023-11-24T08:55:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:55:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:55:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:55:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-11-24T08:55:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_24.status 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_24.status 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_24.status 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_24.status 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_24.status 2023-11-24T08:55:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:55:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.status 2023-11-24T08:55:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-11-24T08:55:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-11-24T08:55:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 165 2023-11-24T08:55:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_24.status 2023-11-24T08:55:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_24.status 2023-11-24T08:55:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-11-24T08:55:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:55:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=165 2023-11-24T08:55:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:55:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.status 2023-11-24T08:55:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.status 2023-11-24T08:55:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-11-24T08:55:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=91 2023-11-24T08:55:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 165 = 91 2023-11-24T08:55:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:55:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.status 2023-11-24T08:55:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.status 2023-11-24T08:55:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-11-24T08:55:19.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=91 2023-11-24T08:55:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 165 = 91 2023-11-24T08:55:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:55:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.status 2023-11-24T08:55:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.status 2023-11-24T08:55:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-11-24T08:55:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:55:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:55:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:55:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:55:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 165 = 2023-11-24T08:55:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:55:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.status 2023-11-24T08:55:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.status 2023-11-24T08:55:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-11-24T08:55:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=165 2023-11-24T08:55:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 165 = 165 2023-11-24T08:55:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-11-24T08:55:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:22.717 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-24T08:55:22.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-24T08:55:22.718 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-24T08:55:22.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-24T08:55:22.721 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2023-11-24T08:55:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-11-24T08:55:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_24': 2023-11-24T08:55:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:55:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:55:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 129136d217f5 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.129136d217f5 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:09 2023 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:09 2023 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:09 2023 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:55:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:55:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 2238b317-dd07-4472-9062-bd252e8143ec 2023-11-24T08:55:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:55:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_24 snap 2023-11-24T08:55:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T08:55:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:55:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_24.export 2023-11-24T08:55:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.export 2023-11-24T08:55:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_24.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.export 2023-11-24T08:55:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_24.export 2023-11-24T08:55:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.export 2023-11-24T08:55:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_24.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.export 2023-11-24T08:55:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_24.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_24.export 2023-11-24T08:55:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_25 2023-11-24T08:55:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_25 snap 2023-11-24T08:55:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:55:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-11-24T08:55:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-11-24T08:55:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:55:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-11-24T08:55:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:55:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:55:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:55:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-11-24T08:55:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:55:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ cluster=cluster1 2023-11-24T08:55:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:55:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:55:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i+ state 2023-11-24T08:55:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sleep 0 2023-11-24T08:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:55:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:55:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-11-24T08:55:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:55:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:55:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-11-24T08:55:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-11-24T08:55:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-11-24T08:55:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:55:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-11-24T08:55:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-11-24T08:55:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2023-11-24T08:55:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:55:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:55:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:55:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:55:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2023-11-24T08:55:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:55:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:55:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:55:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:55:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:55:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:55:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-11-24T08:55:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_25.status 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_25.status 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_25.status 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_25.status 2023-11-24T08:55:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_25.status 2023-11-24T08:55:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:55:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.status 2023-11-24T08:55:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-11-24T08:55:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-11-24T08:55:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 167 2023-11-24T08:55:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_25.status 2023-11-24T08:55:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_25.status 2023-11-24T08:55:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-11-24T08:55:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:55:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=167 2023-11-24T08:55:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:55:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.status 2023-11-24T08:55:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.status 2023-11-24T08:55:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-11-24T08:55:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:55:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:55:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:55:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:55:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 167 = 2023-11-24T08:55:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:55:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.status 2023-11-24T08:55:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.status 2023-11-24T08:55:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-11-24T08:55:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:55:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:55:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:55:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:55:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 167 = 2023-11-24T08:55:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:55:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.status 2023-11-24T08:55:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.status 2023-11-24T08:55:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-11-24T08:55:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=167 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 167 = 167 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-11-24T08:55:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_25': 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1291926cbfcf 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1291926cbfcf 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:15 2023 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:15 2023 2023-11-24T08:55:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:15 2023 2023-11-24T08:55:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:55:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:55:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:55:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9114b26e-4897-4df7-936d-4ec836b81477 2023-11-24T08:55:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:55:30.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_25 snap 2023-11-24T08:55:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T08:55:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:55:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_25.export 2023-11-24T08:55:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.export 2023-11-24T08:55:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_25.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.export 2023-11-24T08:55:30.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_25.export 2023-11-24T08:55:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.export 2023-11-24T08:55:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_25.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.export 2023-11-24T08:55:31.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_25.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_25.export 2023-11-24T08:55:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_26 2023-11-24T08:55:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_26 snap 2023-11-24T08:55:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:55:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-11-24T08:55:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:55:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:55:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:55:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status+ mirror/image_26 2023-11-24T08:55:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ localgrep -i cluster_instance=cluster1 state 2023-11-24T08:55:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:55:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:55:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:55:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:55:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:55:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:55:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:55:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-11-24T08:55:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-11-24T08:55:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-11-24T08:55:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:55:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:55:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:55:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-11-24T08:55:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-11-24T08:55:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2023-11-24T08:55:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:55:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:55:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:55:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:55:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:55:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:55:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2023-11-24T08:55:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:55:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:55:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:55:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:55:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:55:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:55:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-11-24T08:55:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_26.status 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_26.status 2023-11-24T08:55:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:55:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_26.status 2023-11-24T08:55:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_26.status 2023-11-24T08:55:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_26.status 2023-11-24T08:55:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:55:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.status 2023-11-24T08:55:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-11-24T08:55:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-11-24T08:55:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 169 2023-11-24T08:55:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_26.status 2023-11-24T08:55:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_26.status 2023-11-24T08:55:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-11-24T08:55:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:55:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=169 2023-11-24T08:55:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:34.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:55:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.status 2023-11-24T08:55:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.status 2023-11-24T08:55:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-11-24T08:55:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=93 2023-11-24T08:55:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 169 = 93 2023-11-24T08:55:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:55:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.status 2023-11-24T08:55:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.status 2023-11-24T08:55:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-11-24T08:55:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:55:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:55:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:55:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:55:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 169 = 2023-11-24T08:55:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:55:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.status 2023-11-24T08:55:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.status 2023-11-24T08:55:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-11-24T08:55:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:55:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:55:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:55:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:55:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:55:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 169 = 2023-11-24T08:55:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:55:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.status 2023-11-24T08:55:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.status 2023-11-24T08:55:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-11-24T08:55:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:55:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=169 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 169 = 169 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-11-24T08:55:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_26': 2023-11-24T08:55:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:55:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:55:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:55:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1291fcc97ea1 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1291fcc97ea1 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:25 2023 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:25 2023 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:25 2023 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: e79a8072-8f71-4674-9961-be59755f749a 2023-11-24T08:55:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:55:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_26 snap 2023-11-24T08:55:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T08:55:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:55:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_26.export 2023-11-24T08:55:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.export 2023-11-24T08:55:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_26.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.export 2023-11-24T08:55:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_26.export 2023-11-24T08:55:39.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.export 2023-11-24T08:55:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:55:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_26.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.export 2023-11-24T08:55:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_26.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_26.export 2023-11-24T08:55:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:55:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_27 2023-11-24T08:55:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_27 snap 2023-11-24T08:55:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:55:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:55:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:55:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:55:40.629 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2568471941692162) > weight (0.25) 2023-11-24T08:55:40.629 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-24T08:55:40.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-24T08:55:40.630 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-11-24T08:55:40.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T08:55:40.629+0000 7f171913c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 55853) UID: 1000 2023-11-24T08:55:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-11-24T08:55:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:55:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-11-24T08:55:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:55:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:55:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:55:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-11-24T08:55:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:55:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:55:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:55:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:55:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:55:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-11-24T08:55:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:55:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + shift 2023-11-24T08:55:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state+ 2023-11-24T08:55:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:local instance=cluster1 2023-11-24T08:55:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:55:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:55:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:55:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:55:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-11-24T08:55:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:55:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:55:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:55:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-11-24T08:55:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-11-24T08:55:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:55:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:55:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:55:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:55:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:55:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:55:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-11-24T08:55:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-11-24T08:55:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-11-24T08:55:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:55:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:55:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:55:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-11-24T08:55:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2023-11-24T08:55:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:55:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:55:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2023-11-24T08:55:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:55:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:55:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:55:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-11-24T08:55:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:55:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-11-24T08:55:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:55:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:55:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:55:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:55:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-11-24T08:55:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2023-11-24T08:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:55:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:55:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2023-11-24T08:55:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:55:44.631 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-24T08:55:44.631 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:55:44.632 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-24T08:55:44.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-24T08:55:44.632 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-24T08:55:44.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-24T08:55:44.635 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2023-11-24T08:55:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:55:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:55:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-11-24T08:55:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:55:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:55:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:55:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-11-24T08:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-11-24T08:55:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:55:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:55:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:55:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:55:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:55:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:45.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-11-24T08:55:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-11-24T08:55:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:55:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:55:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:55:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-11-24T08:55:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2023-11-24T08:55:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:55:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:55:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:55:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:46.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:46.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2023-11-24T08:55:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T08:55:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:55:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:55:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-11-24T08:55:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:55:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:55:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:55:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:55:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-11-24T08:55:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-11-24T08:55:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:55:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:55:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:55:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-11-24T08:55:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-11-24T08:55:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:55:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:55:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:55:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-11-24T08:55:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2023-11-24T08:55:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:55:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2023-11-24T08:55:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:55:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:55:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:55:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-11-24T08:55:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:55:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:55:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:55:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:55:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:55:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:55:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:55:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:55:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:55:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:55:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:55:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-11-24T08:55:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-11-24T08:55:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:55:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:55:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:55:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:55:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:55:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:55:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-11-24T08:55:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-11-24T08:55:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:55:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:55:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:55:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:55:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-11-24T08:55:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2023-11-24T08:55:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:55:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:55:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:55:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:55:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:55:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:55:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:55:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2023-11-24T08:55:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:55:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:55:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:55:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:56:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:56:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:56:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-11-24T08:56:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:56:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:56:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-11-24T08:56:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:56:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-11-24T08:56:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:56:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:56:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:56:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:56:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:56:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:56:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:56:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:56:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-11-24T08:56:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:56:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-11-24T08:56:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:56:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:56:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:56:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:56:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:56:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:56:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:56:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:56:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:56:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:56:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:56:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:56:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:56:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:56:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:56:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-11-24T08:56:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:56:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2023-11-24T08:56:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:56:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:56:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:56:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:56:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:56:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:56:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:56:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:56:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:56:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:56:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:56:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:56:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:56:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:56:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:56:07.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2023-11-24T08:56:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:56:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:56:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:56:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:56:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-11-24T08:56:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:56:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:56:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-11-24T08:56:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-24T08:56:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-11-24T08:56:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:56:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:56:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:56:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:56:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-11-24T08:56:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:56:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:56:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:56:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:56:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:56:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:56:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-11-24T08:56:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_27.status 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_27.status 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_27.status 2023-11-24T08:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_27.status 2023-11-24T08:56:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_27.status 2023-11-24T08:56:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.status 2023-11-24T08:56:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-11-24T08:56:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-11-24T08:56:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 171 2023-11-24T08:56:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_27.status 2023-11-24T08:56:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_27.status 2023-11-24T08:56:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-11-24T08:56:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:56:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=171 2023-11-24T08:56:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:56:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.status 2023-11-24T08:56:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.status 2023-11-24T08:56:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-11-24T08:56:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=94 2023-11-24T08:56:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 171 = 94 2023-11-24T08:56:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:56:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.status 2023-11-24T08:56:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.status 2023-11-24T08:56:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-11-24T08:56:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 171 = 2023-11-24T08:56:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:56:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.status 2023-11-24T08:56:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.status 2023-11-24T08:56:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-11-24T08:56:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 171 = 2023-11-24T08:56:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:56:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.status 2023-11-24T08:56:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.status 2023-11-24T08:56:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-11-24T08:56:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=171 2023-11-24T08:56:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 171 = 171 2023-11-24T08:56:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-11-24T08:56:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-11-24T08:56:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_27': 2023-11-24T08:56:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:56:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:56:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:56:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1287688449b4 2023-11-24T08:56:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1287688449b4 2023-11-24T08:56:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:56:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:56:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:56:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:56:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:38 2023 2023-11-24T08:56:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:38 2023 2023-11-24T08:56:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:38 2023 2023-11-24T08:56:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:56:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:56:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:56:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 170b38bb-5513-4ade-9ac7-4a9823217a98 2023-11-24T08:56:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:56:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_27 snap 2023-11-24T08:56:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T08:56:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:56:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_27.export 2023-11-24T08:56:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.export 2023-11-24T08:56:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_27.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.export 2023-11-24T08:56:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_27.export 2023-11-24T08:56:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.export 2023-11-24T08:56:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_27.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.export 2023-11-24T08:56:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_27.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_27.export 2023-11-24T08:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_28 2023-11-24T08:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_28 snap 2023-11-24T08:56:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:56:23.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-11-24T08:56:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-11-24T08:56:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:23.407 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:56:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-11-24T08:56:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_28 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:56:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:56:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:56:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-11-24T08:56:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:56:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-11-24T08:56:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-11-24T08:56:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-11-24T08:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:56:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-11-24T08:56:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-11-24T08:56:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2023-11-24T08:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:56:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:56:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:56:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:56:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:56:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2023-11-24T08:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-11-24T08:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2023-11-24T08:56:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:56:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:56:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:56:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:56:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:56:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2023-11-24T08:56:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:56:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:56:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:56:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:56:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:56:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:56:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-11-24T08:56:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:56:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:56:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-11-24T08:56:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_28.status 2023-11-24T08:56:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_28.status 2023-11-24T08:56:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_28.status 2023-11-24T08:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_28.status 2023-11-24T08:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_28.status 2023-11-24T08:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.status 2023-11-24T08:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-11-24T08:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-11-24T08:56:25.416 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 173 2023-11-24T08:56:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_28.status 2023-11-24T08:56:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_28.status 2023-11-24T08:56:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-11-24T08:56:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:56:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=173 2023-11-24T08:56:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:56:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.status 2023-11-24T08:56:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.status 2023-11-24T08:56:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-11-24T08:56:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 173 = 2023-11-24T08:56:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:56:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.status 2023-11-24T08:56:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.status 2023-11-24T08:56:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-11-24T08:56:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 173 = 2023-11-24T08:56:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:56:26.662 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T08:56:26.660+0000 7fee2f6de640 -1 reset not still connected to 0x55a91d92bc70 2023-11-24T08:56:26.663 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T08:56:26.660+0000 7fee2f6de640 -1 reset not still connected to 0x55a95a95bad0 2023-11-24T08:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.status 2023-11-24T08:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.status 2023-11-24T08:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-11-24T08:56:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=173 2023-11-24T08:56:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 173 = 173 2023-11-24T08:56:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-11-24T08:56:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_28': 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12911d5edfab 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12911d5edfab 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:48 2023 2023-11-24T08:56:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:48 2023 2023-11-24T08:56:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:48 2023 2023-11-24T08:56:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:56:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:56:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:56:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: dd54726b-38b5-4534-a5b9-b85acdcdf924 2023-11-24T08:56:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:56:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_28 snap 2023-11-24T08:56:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T08:56:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:56:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_28.export 2023-11-24T08:56:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.export 2023-11-24T08:56:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_28.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.export 2023-11-24T08:56:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_28.export 2023-11-24T08:56:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.export 2023-11-24T08:56:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_28.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.export 2023-11-24T08:56:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_28.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_28.export 2023-11-24T08:56:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_29 2023-11-24T08:56:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_29 snap 2023-11-24T08:56:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:56:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-11-24T08:56:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-11-24T08:56:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:56:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-11-24T08:56:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:56:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:56:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:56:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-11-24T08:56:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:56:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:56:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i state 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:56:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:56:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-11-24T08:56:31.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:56:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:56:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:56:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:56:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:56:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:56:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-11-24T08:56:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_29.status 2023-11-24T08:56:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_29.status 2023-11-24T08:56:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_29.status 2023-11-24T08:56:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_29.status 2023-11-24T08:56:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_29.status 2023-11-24T08:56:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.status 2023-11-24T08:56:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-11-24T08:56:31.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-11-24T08:56:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 175 2023-11-24T08:56:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_29.status 2023-11-24T08:56:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_29.status 2023-11-24T08:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-11-24T08:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:56:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=175 2023-11-24T08:56:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:56:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.status 2023-11-24T08:56:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.status 2023-11-24T08:56:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-11-24T08:56:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=96 2023-11-24T08:56:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 175 = 96 2023-11-24T08:56:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:56:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.status 2023-11-24T08:56:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.status 2023-11-24T08:56:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-11-24T08:56:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 175 = 2023-11-24T08:56:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:56:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.status 2023-11-24T08:56:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.status 2023-11-24T08:56:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-11-24T08:56:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 175 = 2023-11-24T08:56:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:56:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.status 2023-11-24T08:56:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.status 2023-11-24T08:56:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-11-24T08:56:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=175 2023-11-24T08:56:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 175 = 175 2023-11-24T08:56:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-11-24T08:56:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-11-24T08:56:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_29': 2023-11-24T08:56:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:56:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:56:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:56:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 11c1201e13fd 2023-11-24T08:56:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.11c1201e13fd 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:46:55 2023 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:46:55 2023 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:46:55 2023 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 9ed4d512-1473-4b48-9cd4-c90c2f60808e 2023-11-24T08:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:56:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_29 snap 2023-11-24T08:56:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T08:56:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:56:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_29.export 2023-11-24T08:56:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.export 2023-11-24T08:56:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_29.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.export 2023-11-24T08:56:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_29.export 2023-11-24T08:56:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.export 2023-11-24T08:56:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_29.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.export 2023-11-24T08:56:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_29.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_29.export 2023-11-24T08:56:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_30 2023-11-24T08:56:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_30 snap 2023-11-24T08:56:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:56:38.722 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.32233434162539043) > weight (0.25) 2023-11-24T08:56:38.722 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.862686669658146) > weight (0.25) 2023-11-24T08:56:38.722 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3828311958816244) > weight (0.25) 2023-11-24T08:56:38.722 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7274215495095664) > weight (0.25) 2023-11-24T08:56:38.722 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-11-24T08:56:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-11-24T08:56:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-11-24T08:56:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:56:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-11-24T08:56:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:56:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:56:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:56:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_30+ 2023-11-24T08:56:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1grep 2023-11-24T08:56:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -i state 2023-11-24T08:56:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:56:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:56:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:56:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:56:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:56:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-11-24T08:56:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:56:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:56:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-11-24T08:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:56:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:56:40.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:56:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:56:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:56:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-11-24T08:56:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_30.status 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_30.status 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_30.status 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_30.status 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_30.status 2023-11-24T08:56:40.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.status 2023-11-24T08:56:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-11-24T08:56:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-11-24T08:56:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 177 2023-11-24T08:56:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_30.status 2023-11-24T08:56:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_30.status 2023-11-24T08:56:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-11-24T08:56:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:56:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=177 2023-11-24T08:56:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:56:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.status 2023-11-24T08:56:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.status 2023-11-24T08:56:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-11-24T08:56:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=97 2023-11-24T08:56:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 177 = 97 2023-11-24T08:56:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:56:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.status 2023-11-24T08:56:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.status 2023-11-24T08:56:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-11-24T08:56:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 177 = 2023-11-24T08:56:42.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:56:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.status 2023-11-24T08:56:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.status 2023-11-24T08:56:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-11-24T08:56:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 177 = 2023-11-24T08:56:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:56:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.status 2023-11-24T08:56:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.status 2023-11-24T08:56:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-11-24T08:56:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=177 2023-11-24T08:56:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 177 = 177 2023-11-24T08:56:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-11-24T08:56:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-11-24T08:56:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_30': 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120725b3428e 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120725b3428e 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:56:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:08 2023 2023-11-24T08:56:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:08 2023 2023-11-24T08:56:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:08 2023 2023-11-24T08:56:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:56:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:56:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:56:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d77de178-524d-4cb8-ba9c-8399123b0719 2023-11-24T08:56:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:56:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_30 snap 2023-11-24T08:56:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T08:56:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:56:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_30.export 2023-11-24T08:56:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.export 2023-11-24T08:56:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_30.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.export 2023-11-24T08:56:46.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_30.export 2023-11-24T08:56:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.export 2023-11-24T08:56:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_30.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.export 2023-11-24T08:56:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_30.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_30.export 2023-11-24T08:56:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_31 2023-11-24T08:56:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_31 snap 2023-11-24T08:56:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:56:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-11-24T08:56:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-11-24T08:56:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:56:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-11-24T08:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:56:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-11-24T08:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster=cluster1 2023-11-24T08:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:56:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep+ -ilocal state loop_instance 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:56:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:56:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-11-24T08:56:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:56:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:56:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:56:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:56:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:56:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-11-24T08:56:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_31.status 2023-11-24T08:56:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_31.status 2023-11-24T08:56:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_31.status 2023-11-24T08:56:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_31.status 2023-11-24T08:56:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_31.status 2023-11-24T08:56:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.status 2023-11-24T08:56:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-11-24T08:56:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-11-24T08:56:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 179 2023-11-24T08:56:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_31.status 2023-11-24T08:56:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_31.status 2023-11-24T08:56:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-11-24T08:56:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:56:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=179 2023-11-24T08:56:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:56:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.status 2023-11-24T08:56:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.status 2023-11-24T08:56:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-11-24T08:56:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=98 2023-11-24T08:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 179 = 98 2023-11-24T08:56:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:56:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.status 2023-11-24T08:56:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.status 2023-11-24T08:56:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-11-24T08:56:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:51.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 179 = 2023-11-24T08:56:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:56:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.status 2023-11-24T08:56:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.status 2023-11-24T08:56:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-11-24T08:56:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:56:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:56:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:56:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:56:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 179 = 2023-11-24T08:56:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:56:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.status 2023-11-24T08:56:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.status 2023-11-24T08:56:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-11-24T08:56:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:56:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=179 2023-11-24T08:56:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 179 = 179 2023-11-24T08:56:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-11-24T08:56:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_31': 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12a0e624f506 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12a0e624f506 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:56:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:56:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:14 2023 2023-11-24T08:56:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:14 2023 2023-11-24T08:56:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:14 2023 2023-11-24T08:56:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:56:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:56:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:56:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 736a6ea1-a614-4fbc-8e6f-f5a28a8d570f 2023-11-24T08:56:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:56:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_31 snap 2023-11-24T08:56:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T08:56:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_31.export 2023-11-24T08:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.export 2023-11-24T08:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_31.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.export 2023-11-24T08:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_31.export 2023-11-24T08:56:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.export 2023-11-24T08:56:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:56:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_31.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.export 2023-11-24T08:56:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_31.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_31.export 2023-11-24T08:56:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_32 2023-11-24T08:56:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_32 snap 2023-11-24T08:56:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:56:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:56:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:56:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:56:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i+ state 2023-11-24T08:56:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:shift 2023-11-24T08:56:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:56:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:56:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:56:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:56:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:56:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-11-24T08:56:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:56:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:56:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:56:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:56:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:56:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:56:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:56:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:56:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:56:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:56:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:56:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-11-24T08:56:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-11-24T08:56:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-11-24T08:56:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:56:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:56:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:56:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:56:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:56:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:56:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:56:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:56:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:56:58.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-11-24T08:56:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-11-24T08:56:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2023-11-24T08:56:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:56:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:56:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:56:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:56:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:56:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:56:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2023-11-24T08:56:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-11-24T08:56:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:56:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:56:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2023-11-24T08:56:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:56:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:56:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:56:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:56:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:56:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:56:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:56:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:56:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:56:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:56:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:56:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2023-11-24T08:56:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:56:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:56:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:56:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:56:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:56:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:56:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_32.status 2023-11-24T08:56:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_32.status 2023-11-24T08:56:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_32.status 2023-11-24T08:56:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_32.status 2023-11-24T08:56:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_32.status 2023-11-24T08:56:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:56:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.status 2023-11-24T08:56:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-11-24T08:56:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:56:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-11-24T08:56:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 181 2023-11-24T08:56:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_32.status 2023-11-24T08:56:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:56:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:56:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:56:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_32.status 2023-11-24T08:56:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-11-24T08:56:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:56:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:56:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=181 2023-11-24T08:56:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:56:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:57:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.status 2023-11-24T08:57:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:57:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.status 2023-11-24T08:57:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-11-24T08:57:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:00.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=99 2023-11-24T08:57:00.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 181 = 99 2023-11-24T08:57:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:57:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.status 2023-11-24T08:57:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:57:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.status 2023-11-24T08:57:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-11-24T08:57:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 181 = 2023-11-24T08:57:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:57:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.status 2023-11-24T08:57:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:57:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.status 2023-11-24T08:57:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-11-24T08:57:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 181 = 2023-11-24T08:57:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:57:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.status 2023-11-24T08:57:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:57:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.status 2023-11-24T08:57:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-11-24T08:57:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=181 2023-11-24T08:57:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 181 = 181 2023-11-24T08:57:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-11-24T08:57:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:57:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-11-24T08:57:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_32': 2023-11-24T08:57:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:57:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12a0a8e13817 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12a0a8e13817 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:28 2023 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:28 2023 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:28 2023 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:57:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:57:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:57:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: fea5c78b-14c3-43da-93a2-d38278c1541a 2023-11-24T08:57:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:57:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_32 snap 2023-11-24T08:57:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T08:57:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:57:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_32.export 2023-11-24T08:57:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.export 2023-11-24T08:57:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_32.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.export 2023-11-24T08:57:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_32.export 2023-11-24T08:57:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.export 2023-11-24T08:57:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_32.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.export 2023-11-24T08:57:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_32.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_32.export 2023-11-24T08:57:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_33 2023-11-24T08:57:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_33 snap 2023-11-24T08:57:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:57:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-11-24T08:57:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:57:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:57:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:57:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-11-24T08:57:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:57:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + shift 2023-11-24T08:57:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:57:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:57:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:57:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:57:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:57:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:57:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:57:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-11-24T08:57:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-11-24T08:57:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-11-24T08:57:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:57:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:08.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-11-24T08:57:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-11-24T08:57:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:57:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:57:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2023-11-24T08:57:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:57:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + grep -i Replaying 2023-11-24T08:57:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo "state": "Replaying" 2023-11-24T08:57:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:57:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:57:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:57:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-11-24T08:57:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:57:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:57:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-11-24T08:57:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_33.status 2023-11-24T08:57:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_33.status 2023-11-24T08:57:08.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_33.status 2023-11-24T08:57:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_33.status 2023-11-24T08:57:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_33.status 2023-11-24T08:57:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.status 2023-11-24T08:57:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-11-24T08:57:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-11-24T08:57:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 183 2023-11-24T08:57:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_33.status 2023-11-24T08:57:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_33.status 2023-11-24T08:57:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-11-24T08:57:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:57:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=183 2023-11-24T08:57:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:57:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.status 2023-11-24T08:57:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.status 2023-11-24T08:57:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-11-24T08:57:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=100 2023-11-24T08:57:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 183 = 100 2023-11-24T08:57:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:57:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.status 2023-11-24T08:57:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.status 2023-11-24T08:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-11-24T08:57:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 183 = 2023-11-24T08:57:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:57:10.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.status 2023-11-24T08:57:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.status 2023-11-24T08:57:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-11-24T08:57:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 183 = 2023-11-24T08:57:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:57:12.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.status 2023-11-24T08:57:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.status 2023-11-24T08:57:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-11-24T08:57:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=183 2023-11-24T08:57:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 183 = 183 2023-11-24T08:57:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-11-24T08:57:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-11-24T08:57:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_33': 2023-11-24T08:57:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:57:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:57:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12078eb8470c 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12078eb8470c 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:43 2023 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:43 2023 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:43 2023 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:57:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:57:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 6956a2e8-c6e1-4b1f-8188-481ec5f6a81b 2023-11-24T08:57:13.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:57:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_33 snap 2023-11-24T08:57:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T08:57:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:57:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_33.export 2023-11-24T08:57:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.export 2023-11-24T08:57:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_33.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.export 2023-11-24T08:57:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_33.export 2023-11-24T08:57:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.export 2023-11-24T08:57:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_33.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.export 2023-11-24T08:57:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_33.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_33.export 2023-11-24T08:57:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_34 2023-11-24T08:57:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_34 snap 2023-11-24T08:57:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:57:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-11-24T08:57:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-11-24T08:57:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:57:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-11-24T08:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:57:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:57:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:57:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-11-24T08:57:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:57:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + shift 2023-11-24T08:57:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:57:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:57:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:57:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:57:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:57:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:57:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-11-24T08:57:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-11-24T08:57:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-11-24T08:57:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:57:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:57:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:57:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:57:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-11-24T08:57:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:57:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:57:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:57:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:57:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_34.status 2023-11-24T08:57:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_34.status 2023-11-24T08:57:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_34.status 2023-11-24T08:57:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_34.status 2023-11-24T08:57:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_34.status 2023-11-24T08:57:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.status 2023-11-24T08:57:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-11-24T08:57:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-11-24T08:57:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 185 2023-11-24T08:57:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_34.status 2023-11-24T08:57:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_34.status 2023-11-24T08:57:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-11-24T08:57:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:57:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=185 2023-11-24T08:57:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:57:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.status 2023-11-24T08:57:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.status 2023-11-24T08:57:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-11-24T08:57:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=101 2023-11-24T08:57:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 185 = 101 2023-11-24T08:57:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:57:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.status 2023-11-24T08:57:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.status 2023-11-24T08:57:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-11-24T08:57:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:18.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:18.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 185 = 2023-11-24T08:57:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:57:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.status 2023-11-24T08:57:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.status 2023-11-24T08:57:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-11-24T08:57:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 185 = 2023-11-24T08:57:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:57:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.status 2023-11-24T08:57:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.status 2023-11-24T08:57:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-11-24T08:57:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=185 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 185 = 185 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-11-24T08:57:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_34': 2023-11-24T08:57:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:57:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1287690abb 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1287690abb 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:47:58 2023 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:47:58 2023 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:47:58 2023 2023-11-24T08:57:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:57:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:57:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:57:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 3d5be2fc-e83f-4e85-8d39-4fe13e555474 2023-11-24T08:57:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:57:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_34 snap 2023-11-24T08:57:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T08:57:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:57:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_34.export 2023-11-24T08:57:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.export 2023-11-24T08:57:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_34.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.export 2023-11-24T08:57:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_34.export 2023-11-24T08:57:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.export 2023-11-24T08:57:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_34.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.export 2023-11-24T08:57:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_34.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_34.export 2023-11-24T08:57:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_35 2023-11-24T08:57:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_35 snap 2023-11-24T08:57:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:57:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-11-24T08:57:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:57:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:57:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:57:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-11-24T08:57:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:57:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ cluster=cluster1 2023-11-24T08:57:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:57:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ localgrep loop_instance 2023-11-24T08:57:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -i state 2023-11-24T08:57:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:57:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:57:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:57:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-11-24T08:57:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:57:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-11-24T08:57:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:57:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:57:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:57:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:57:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:57:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:57:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-11-24T08:57:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_35.status 2023-11-24T08:57:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_35.status 2023-11-24T08:57:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_35.status 2023-11-24T08:57:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_35.status 2023-11-24T08:57:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_35.status 2023-11-24T08:57:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.status 2023-11-24T08:57:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-11-24T08:57:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-11-24T08:57:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 187 2023-11-24T08:57:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_35.status 2023-11-24T08:57:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_35.status 2023-11-24T08:57:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-11-24T08:57:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:57:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=187 2023-11-24T08:57:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:57:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.status 2023-11-24T08:57:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.status 2023-11-24T08:57:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-11-24T08:57:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=102 2023-11-24T08:57:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 187 = 102 2023-11-24T08:57:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:57:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.status 2023-11-24T08:57:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.status 2023-11-24T08:57:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-11-24T08:57:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=102 2023-11-24T08:57:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 187 = 102 2023-11-24T08:57:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:57:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.status 2023-11-24T08:57:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.status 2023-11-24T08:57:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-11-24T08:57:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 187 = 2023-11-24T08:57:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:57:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.status 2023-11-24T08:57:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.status 2023-11-24T08:57:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-11-24T08:57:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=187 2023-11-24T08:57:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 187 = 187 2023-11-24T08:57:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-11-24T08:57:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-11-24T08:57:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_35': 2023-11-24T08:57:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:57:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:57:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:57:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1291679feceb 2023-11-24T08:57:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1291679feceb 2023-11-24T08:57:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:11 2023 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:11 2023 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:11 2023 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: c8458320-3838-4775-b00c-eeaf0f63bb66 2023-11-24T08:57:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:57:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_35 snap 2023-11-24T08:57:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T08:57:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:57:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_35.export 2023-11-24T08:57:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.export 2023-11-24T08:57:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_35.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.export 2023-11-24T08:57:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_35.export 2023-11-24T08:57:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.export 2023-11-24T08:57:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_35.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.export 2023-11-24T08:57:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_35.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_35.export 2023-11-24T08:57:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_36 2023-11-24T08:57:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_36 snap 2023-11-24T08:57:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:57:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-11-24T08:57:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-11-24T08:57:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:57:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:33.755 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:57:33.751+0000 7f42b0626640 -1 reset not still connected to 0x561a522b6750 2023-11-24T08:57:33.755 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:57:33.751+0000 7f42b0626640 -1 reset not still connected to 0x561a522b6c30 2023-11-24T08:57:33.758 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:57:33.751+0000 7f42b0626640 -1 reset not still connected to 0x561a522b6f70 2023-11-24T08:57:33.758 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:57:33.751+0000 7f42b0626640 -1 reset not still connected to 0x561a582d1380 2023-11-24T08:57:33.760 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T08:57:33.755+0000 7f42b0626640 -1 reset not still connected to 0x561a65fc1c70 2023-11-24T08:57:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-11-24T08:57:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:57:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:57:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:57:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_36 2023-11-24T08:57:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:57:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:57:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:57:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:57:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:57:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:57:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:57:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-11-24T08:57:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:57:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:57:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:57:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-11-24T08:57:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:57:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:57:34.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:57:34.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:57:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:57:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:57:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-11-24T08:57:34.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_36.status 2023-11-24T08:57:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_36.status 2023-11-24T08:57:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_36.status 2023-11-24T08:57:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_36.status 2023-11-24T08:57:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_36.status 2023-11-24T08:57:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.status 2023-11-24T08:57:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-11-24T08:57:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-11-24T08:57:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 189 2023-11-24T08:57:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_36.status 2023-11-24T08:57:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_36.status 2023-11-24T08:57:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-11-24T08:57:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:57:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=189 2023-11-24T08:57:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:57:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.status 2023-11-24T08:57:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.status 2023-11-24T08:57:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-11-24T08:57:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 189 = 2023-11-24T08:57:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:57:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.status 2023-11-24T08:57:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.status 2023-11-24T08:57:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-11-24T08:57:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 189 = 2023-11-24T08:57:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:57:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.status 2023-11-24T08:57:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.status 2023-11-24T08:57:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36+ --format xml 2023-11-24T08:57:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 189 = 2023-11-24T08:57:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:57:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.status 2023-11-24T08:57:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.status 2023-11-24T08:57:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-11-24T08:57:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=189 2023-11-24T08:57:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 189 = 189 2023-11-24T08:57:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-11-24T08:57:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-11-24T08:57:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_36': 2023-11-24T08:57:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:57:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12a07a379576 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12a07a379576 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:26 2023 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:26 2023 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:26 2023 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:57:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:57:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: fc86b320-d291-4bfa-a035-067b30e073fa 2023-11-24T08:57:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:57:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_36 snap 2023-11-24T08:57:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T08:57:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:57:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_36.export 2023-11-24T08:57:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.export 2023-11-24T08:57:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_36.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.export 2023-11-24T08:57:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_36.export 2023-11-24T08:57:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:40.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.export 2023-11-24T08:57:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_36.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.export 2023-11-24T08:57:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_36.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_36.export 2023-11-24T08:57:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_37 2023-11-24T08:57:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_37 snap 2023-11-24T08:57:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:57:40.636 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.41651260524384903) > weight (0.25) 2023-11-24T08:57:40.636 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-24T08:57:40.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-24T08:57:40.637 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-11-24T08:57:40.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T08:57:40.635+0000 7f7943896640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 60318) UID: 1000 2023-11-24T08:57:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-11-24T08:57:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-11-24T08:57:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:57:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-11-24T08:57:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:57:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:57:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_37 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:57:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:57:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:57:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-11-24T08:57:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-11-24T08:57:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:43.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-11-24T08:57:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:57:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:57:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:57:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:57:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:57:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:57:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-11-24T08:57:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-11-24T08:57:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2023-11-24T08:57:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:57:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:57:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:57:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:57:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2023-11-24T08:57:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:57:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:57:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:57:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:57:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:57:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:57:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-11-24T08:57:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_37.status 2023-11-24T08:57:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_37.status 2023-11-24T08:57:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_37.status 2023-11-24T08:57:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_37.status 2023-11-24T08:57:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_37.status 2023-11-24T08:57:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:57:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.status 2023-11-24T08:57:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-11-24T08:57:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-11-24T08:57:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 191 2023-11-24T08:57:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_37.status 2023-11-24T08:57:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_37.status 2023-11-24T08:57:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-11-24T08:57:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:57:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=191 2023-11-24T08:57:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:44.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:57:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.status 2023-11-24T08:57:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.status 2023-11-24T08:57:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-11-24T08:57:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=104 2023-11-24T08:57:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 191 = 104 2023-11-24T08:57:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:57:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.status 2023-11-24T08:57:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.status 2023-11-24T08:57:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-11-24T08:57:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=104 2023-11-24T08:57:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 191 = 104 2023-11-24T08:57:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:57:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.status 2023-11-24T08:57:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.status 2023-11-24T08:57:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-11-24T08:57:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:57:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:57:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:57:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:57:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:57:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 191 = 2023-11-24T08:57:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:57:46.638 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-24T08:57:46.638 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:57:46.638 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-24T08:57:46.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-24T08:57:46.638 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-24T08:57:46.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-24T08:57:46.641 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2023-11-24T08:57:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.status 2023-11-24T08:57:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.status 2023-11-24T08:57:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-11-24T08:57:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:57:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:57:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=191 2023-11-24T08:57:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 191 = 191 2023-11-24T08:57:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-11-24T08:57:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-11-24T08:57:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_37': 2023-11-24T08:57:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:57:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:57:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:57:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 120756126e45 2023-11-24T08:57:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.120756126e45 2023-11-24T08:57:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:57:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:57:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:57:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:57:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:34 2023 2023-11-24T08:57:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:34 2023 2023-11-24T08:57:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:34 2023 2023-11-24T08:57:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:57:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:57:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:57:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: cb1bf2ef-bc24-41f4-86ea-0b0c7b483a22 2023-11-24T08:57:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:57:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_37 snap 2023-11-24T08:57:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T08:57:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:57:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_37.export 2023-11-24T08:57:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.export 2023-11-24T08:57:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_37.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.export 2023-11-24T08:57:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_37.export 2023-11-24T08:57:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.export 2023-11-24T08:57:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:57:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_37.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.export 2023-11-24T08:57:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_37.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_37.export 2023-11-24T08:57:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:57:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_38 2023-11-24T08:57:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_38 snap 2023-11-24T08:57:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:57:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:57:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:57:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:57:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-11-24T08:57:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:57:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-11-24T08:57:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:57:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:57:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:57:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-11-24T08:57:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:57:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:57:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:57:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:57:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:57:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-11-24T08:57:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:57:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1+ 2023-11-24T08:57:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:57:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:57:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-11-24T08:57:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:57:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-11-24T08:57:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2023-11-24T08:57:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:57:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:57:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:53.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2023-11-24T08:57:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2023-11-24T08:57:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:57:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:57:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:57:53.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:57:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:57:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:57:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:57:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:57:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2023-11-24T08:57:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:57:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:57:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:57:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-11-24T08:57:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:57:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-11-24T08:57:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-11-24T08:57:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:57:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:57:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-11-24T08:57:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2023-11-24T08:57:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:57:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2023-11-24T08:57:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2023-11-24T08:57:54.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:57:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:57:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:57:54.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:57:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:57:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:57:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2023-11-24T08:57:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:57:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:57:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:57:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-11-24T08:57:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:57:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:57:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:57:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:57:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:57:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-11-24T08:57:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-11-24T08:57:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:57:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:57:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:57:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:57:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:57:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:57:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:57:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:57:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:57:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:57:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:57:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-11-24T08:57:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2023-11-24T08:57:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:57:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:57:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:57:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:57:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:57:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:57:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:57:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:57:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:57:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:57:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:57:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2023-11-24T08:57:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:57:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:57:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:57:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:57:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:57:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:57:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:57:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:57:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:57:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:57:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:57:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2023-11-24T08:57:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:57:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:57:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:57:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T08:58:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:58:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:58:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-11-24T08:58:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:58:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:01.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:58:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-11-24T08:58:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:58:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-11-24T08:58:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:58:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:58:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:58:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:58:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:58:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:58:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-11-24T08:58:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:58:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:58:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:58:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:58:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2023-11-24T08:58:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:58:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_38 2023-11-24T08:58:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:58:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:58:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:58:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:58:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:58:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:58:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:58:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:58:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_38 2023-11-24T08:58:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:58:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:58:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:58:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:58:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-11-24T08:58:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:58:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:58:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-11-24T08:58:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:58:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-11-24T08:58:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:58:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-11-24T08:58:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-24T08:58:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:58:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:58:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:58:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2023-11-24T08:58:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:58:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:58:10.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:58:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:58:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:58:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:58:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-11-24T08:58:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:58:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:58:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:58:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-11-24T08:58:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:58:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_38.status 2023-11-24T08:58:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_38.status 2023-11-24T08:58:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_38.status 2023-11-24T08:58:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_38.status 2023-11-24T08:58:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_38.status 2023-11-24T08:58:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.status 2023-11-24T08:58:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-11-24T08:58:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:58:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-11-24T08:58:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 193 2023-11-24T08:58:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_38.status 2023-11-24T08:58:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:58:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_38.status 2023-11-24T08:58:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-11-24T08:58:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:58:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=193 2023-11-24T08:58:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:58:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.status 2023-11-24T08:58:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:58:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.status 2023-11-24T08:58:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-11-24T08:58:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 193 = 2023-11-24T08:58:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:58:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.status 2023-11-24T08:58:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:58:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.status 2023-11-24T08:58:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-11-24T08:58:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 193 = 2023-11-24T08:58:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:58:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.status 2023-11-24T08:58:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:58:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.status 2023-11-24T08:58:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-11-24T08:58:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=193 2023-11-24T08:58:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 193 = 193 2023-11-24T08:58:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-11-24T08:58:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:58:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:13.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9994738254964964) > weight (0.25) 2023-11-24T08:58:13.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5636309329076055) > weight (0.25) 2023-11-24T08:58:13.723 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-11-24T08:58:13.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-11-24T08:58:13.723 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-11-24T08:58:13.724 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-11-24T08:58:13.724 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:58:13.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T08:58:13.719+0000 7f2e42de7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 51000) UID: 1000 2023-11-24T08:58:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-11-24T08:58:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_38': 2023-11-24T08:58:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:58:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:58:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:58:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1287e9c5b8df 2023-11-24T08:58:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1287e9c5b8df 2023-11-24T08:58:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:58:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:41 2023 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:41 2023 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:41 2023 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 3b2fce77-b38d-48ea-b5c5-cb78f30c2412 2023-11-24T08:58:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:58:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_38 snap 2023-11-24T08:58:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T08:58:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:58:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_38.export 2023-11-24T08:58:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.export 2023-11-24T08:58:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_38.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.export 2023-11-24T08:58:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_38.export 2023-11-24T08:58:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.export 2023-11-24T08:58:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_38.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.export 2023-11-24T08:58:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_38.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_38.export 2023-11-24T08:58:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_39 2023-11-24T08:58:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_39 snap 2023-11-24T08:58:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:58:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-11-24T08:58:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-11-24T08:58:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:58:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-11-24T08:58:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:58:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:58:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:58:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-11-24T08:58:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:58:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster=cluster1 2023-11-24T08:58:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:58:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ greplocal loop_instance 2023-11-24T08:58:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -i state 2023-11-24T08:58:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:58:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:58:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:58:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-11-24T08:58:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:58:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-11-24T08:58:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-11-24T08:58:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-11-24T08:58:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:58:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:58:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:58:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:58:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:58:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:58:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-11-24T08:58:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:58:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:58:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-11-24T08:58:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_39.status 2023-11-24T08:58:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_39.status 2023-11-24T08:58:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_39.status 2023-11-24T08:58:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_39.status 2023-11-24T08:58:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_39.status 2023-11-24T08:58:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.status 2023-11-24T08:58:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-11-24T08:58:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-11-24T08:58:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 195 2023-11-24T08:58:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_39.status 2023-11-24T08:58:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_39.status 2023-11-24T08:58:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-11-24T08:58:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:58:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=195 2023-11-24T08:58:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:58:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.status 2023-11-24T08:58:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.status 2023-11-24T08:58:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-11-24T08:58:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=106 2023-11-24T08:58:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 195 = 106 2023-11-24T08:58:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:58:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.status 2023-11-24T08:58:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.status 2023-11-24T08:58:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-11-24T08:58:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 195 = 2023-11-24T08:58:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:58:19.825 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-11-24T08:58:19.825 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-11-24T08:58:19.826 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-11-24T08:58:19.829 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-11-24T08:58:19.829 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-11-24T08:58:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.status 2023-11-24T08:58:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.status 2023-11-24T08:58:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-11-24T08:58:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 195 = 2023-11-24T08:58:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:58:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.status 2023-11-24T08:58:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.status 2023-11-24T08:58:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-11-24T08:58:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=195 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 195 = 195 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-11-24T08:58:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_39': 2023-11-24T08:58:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:58:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 129169d49754 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.129169d49754 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:48:53 2023 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:48:53 2023 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:48:53 2023 2023-11-24T08:58:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:58:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:58:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:58:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 344f7f3d-a4fa-452d-94e2-e36d500bd26a 2023-11-24T08:58:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:58:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_39 snap 2023-11-24T08:58:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T08:58:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:58:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_39.export 2023-11-24T08:58:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.export 2023-11-24T08:58:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_39.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.export 2023-11-24T08:58:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_39.export 2023-11-24T08:58:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.export 2023-11-24T08:58:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_39.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.export 2023-11-24T08:58:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_39.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_39.export 2023-11-24T08:58:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_40 2023-11-24T08:58:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_40 snap 2023-11-24T08:58:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:58:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-11-24T08:58:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-11-24T08:58:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:58:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-11-24T08:58:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:58:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:58:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:58:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-11-24T08:58:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ cluster_instance=cluster1 2023-11-24T08:58:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:58:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:58:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:58:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:58:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-11-24T08:58:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:58:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:58:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:58:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:58:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:58:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:58:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-11-24T08:58:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:58:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:58:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-11-24T08:58:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_40.status 2023-11-24T08:58:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_40.status 2023-11-24T08:58:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_40.status 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_40.status 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_40.status 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.status 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-11-24T08:58:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 197 2023-11-24T08:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_40.status 2023-11-24T08:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_40.status 2023-11-24T08:58:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-11-24T08:58:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:27.848 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:58:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=197 2023-11-24T08:58:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:58:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.status 2023-11-24T08:58:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.status 2023-11-24T08:58:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-11-24T08:58:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=107 2023-11-24T08:58:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 197 = 107 2023-11-24T08:58:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:58:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.status 2023-11-24T08:58:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.status 2023-11-24T08:58:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-11-24T08:58:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 197 = 2023-11-24T08:58:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:58:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.status 2023-11-24T08:58:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.status 2023-11-24T08:58:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-11-24T08:58:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 197 = 2023-11-24T08:58:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:58:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.status 2023-11-24T08:58:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.status 2023-11-24T08:58:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-11-24T08:58:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:31.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=197 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 197 = 197 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_40': 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12a01260e032 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12a01260e032 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:58:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:00 2023 2023-11-24T08:58:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:00 2023 2023-11-24T08:58:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:00 2023 2023-11-24T08:58:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:58:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:58:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:58:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: b34477ff-894a-40fc-ba47-b95068f79b4c 2023-11-24T08:58:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:58:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_40 snap 2023-11-24T08:58:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T08:58:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:58:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_40.export 2023-11-24T08:58:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.export 2023-11-24T08:58:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_40.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.export 2023-11-24T08:58:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_40.export 2023-11-24T08:58:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.export 2023-11-24T08:58:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_40.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.export 2023-11-24T08:58:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_40.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_40.export 2023-11-24T08:58:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_41 2023-11-24T08:58:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_41 snap 2023-11-24T08:58:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:58:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-11-24T08:58:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-11-24T08:58:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:58:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-11-24T08:58:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:58:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:58:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:58:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons+ cluster1 rbd mirror status mirror/image_41 2023-11-24T08:58:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:58:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:58:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:58:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:58:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:58:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:58:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:58:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-11-24T08:58:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:58:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:58:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:58:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-11-24T08:58:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-11-24T08:58:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-11-24T08:58:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:58:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-11-24T08:58:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-11-24T08:58:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:58:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:58:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2023-11-24T08:58:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:58:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:58:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:58:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:58:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:58:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:58:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-11-24T08:58:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_41.status 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_41.status 2023-11-24T08:58:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_41.status 2023-11-24T08:58:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_41.status 2023-11-24T08:58:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_41.status 2023-11-24T08:58:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.status 2023-11-24T08:58:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-11-24T08:58:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:36.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-11-24T08:58:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 199 2023-11-24T08:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_41.status 2023-11-24T08:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_41.status 2023-11-24T08:58:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-11-24T08:58:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:58:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=199 2023-11-24T08:58:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:58:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.status 2023-11-24T08:58:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.status 2023-11-24T08:58:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-11-24T08:58:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=108 2023-11-24T08:58:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 199 = 108 2023-11-24T08:58:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:58:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.status 2023-11-24T08:58:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.status 2023-11-24T08:58:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-11-24T08:58:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=108 2023-11-24T08:58:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 199 = 108 2023-11-24T08:58:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:58:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.status 2023-11-24T08:58:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.status 2023-11-24T08:58:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-11-24T08:58:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 199 = 2023-11-24T08:58:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:58:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.status 2023-11-24T08:58:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.status 2023-11-24T08:58:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-11-24T08:58:40.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=199 2023-11-24T08:58:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 199 = 199 2023-11-24T08:58:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-11-24T08:58:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-11-24T08:58:41.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_41': 2023-11-24T08:58:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:58:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:58:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:58:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12073481d81c 2023-11-24T08:58:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12073481d81c 2023-11-24T08:58:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:58:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:58:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:58:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:58:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:08 2023 2023-11-24T08:58:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:08 2023 2023-11-24T08:58:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:08 2023 2023-11-24T08:58:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:58:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:58:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:58:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 647ef1ae-5a98-459b-b31b-d5aa0404c570 2023-11-24T08:58:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:58:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_41 snap 2023-11-24T08:58:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T08:58:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:58:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_41.export 2023-11-24T08:58:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.export 2023-11-24T08:58:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_41.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.export 2023-11-24T08:58:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_41.export 2023-11-24T08:58:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.export 2023-11-24T08:58:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_41.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.export 2023-11-24T08:58:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_41.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_41.export 2023-11-24T08:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_42 2023-11-24T08:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_42 snap 2023-11-24T08:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:58:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-11-24T08:58:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:58:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:58:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:58:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-11-24T08:58:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ cluster_instance=cluster1 2023-11-24T08:58:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:58:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:58:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:58:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:58:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:58:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:58:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-11-24T08:58:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-11-24T08:58:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:44.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:44.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:58:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2023-11-24T08:58:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:58:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:58:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:58:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:58:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_42.status 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_42.status 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_42.status 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_42.status 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_42.status 2023-11-24T08:58:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.status 2023-11-24T08:58:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-11-24T08:58:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-11-24T08:58:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 201 2023-11-24T08:58:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_42.status 2023-11-24T08:58:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_42.status 2023-11-24T08:58:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-11-24T08:58:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:58:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=201 2023-11-24T08:58:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:58:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.status 2023-11-24T08:58:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.status 2023-11-24T08:58:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-11-24T08:58:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 201 = 2023-11-24T08:58:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:58:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.status 2023-11-24T08:58:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.status 2023-11-24T08:58:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-11-24T08:58:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 201 = 2023-11-24T08:58:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:58:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.status 2023-11-24T08:58:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.status 2023-11-24T08:58:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-11-24T08:58:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=201 2023-11-24T08:58:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 201 = 201 2023-11-24T08:58:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-11-24T08:58:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-11-24T08:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_42': 2023-11-24T08:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 1287df215cb0 2023-11-24T08:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.1287df215cb0 2023-11-24T08:58:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:17 2023 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:17 2023 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:17 2023 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:58:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: d4241692-b271-471f-97e7-4e41ae0eefd4 2023-11-24T08:58:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:58:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_42 snap 2023-11-24T08:58:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T08:58:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:58:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_42.export 2023-11-24T08:58:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.export 2023-11-24T08:58:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_42.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.export 2023-11-24T08:58:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_42.export 2023-11-24T08:58:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.export 2023-11-24T08:58:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_42.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.export 2023-11-24T08:58:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_42.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_42.export 2023-11-24T08:58:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_43 2023-11-24T08:58:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_43 snap 2023-11-24T08:58:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:58:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-11-24T08:58:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-11-24T08:58:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:58:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-11-24T08:58:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:58:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:58:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:58:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-11-24T08:58:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster_instance=cluster1 2023-11-24T08:58:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T08:58:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:58:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:58:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:58:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:58:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:58:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:58:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-11-24T08:58:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-11-24T08:58:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-11-24T08:58:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T08:58:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T08:58:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T08:58:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T08:58:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T08:58:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T08:58:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T08:58:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-11-24T08:58:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-11-24T08:58:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2023-11-24T08:58:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T08:58:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T08:58:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T08:58:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T08:58:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T08:58:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T08:58:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2023-11-24T08:58:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-11-24T08:58:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T08:58:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T08:58:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_43 2023-11-24T08:58:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T08:58:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:58:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:58:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T08:58:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:58:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:58:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T08:58:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T08:58:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T08:58:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:58:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:58:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T08:58:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_43 2023-11-24T08:58:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:58:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:58:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:58:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:58:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:58:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_43.status 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_43.status 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_43.status 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_43.status 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_43.status 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.status 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-11-24T08:58:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 203 2023-11-24T08:58:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_43.status 2023-11-24T08:58:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_43.status 2023-11-24T08:58:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-11-24T08:58:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:58:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=203 2023-11-24T08:58:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:58:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.status 2023-11-24T08:58:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.status 2023-11-24T08:58:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-11-24T08:58:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=110 2023-11-24T08:58:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 203 = 110 2023-11-24T08:58:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:58:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.status 2023-11-24T08:58:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.status 2023-11-24T08:58:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-11-24T08:58:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 203 = 2023-11-24T08:58:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:58:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.status 2023-11-24T08:58:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.status 2023-11-24T08:58:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-11-24T08:58:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:58:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:58:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:58:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:58:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:58:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 203 = 2023-11-24T08:58:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:58:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.status 2023-11-24T08:58:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.status 2023-11-24T08:58:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-11-24T08:58:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:58:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:58:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=203 2023-11-24T08:58:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 203 = 203 2023-11-24T08:58:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-11-24T08:58:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:58:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-11-24T08:58:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_43': 2023-11-24T08:58:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:58:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:58:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:58:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 129136912b24 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.129136912b24 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:24 2023 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:24 2023 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:24 2023 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:58:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: ab341c38-aeae-40e6-8dbf-7e351d9676c7 2023-11-24T08:58:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:58:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_43 snap 2023-11-24T08:58:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T08:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_43.export 2023-11-24T08:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.export 2023-11-24T08:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_43.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.export 2023-11-24T08:58:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_43.export 2023-11-24T08:58:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:57.408 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:58:57.401+0000 7f4b5a023640 -1 reset not still connected to 0x5602183064e0 2023-11-24T08:58:57.409 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:58:57.405+0000 7f4b5a023640 -1 reset not still connected to 0x560218306750 2023-11-24T08:58:57.409 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:58:57.405+0000 7f4b5a023640 -1 reset not still connected to 0x560218b989c0 2023-11-24T08:58:57.409 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:58:57.405+0000 7f4b5a023640 -1 reset not still connected to 0x560218fbc9c0 2023-11-24T08:58:57.409 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:58:57.405+0000 7f4b5a023640 -1 reset not still connected to 0x5602197a8b60 2023-11-24T08:58:57.409 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:58:57.405+0000 7f4b5a023640 -1 reset not still connected to 0x5602197ab5f0 2023-11-24T08:58:57.410 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:58:57.405+0000 7f4b5a023640 -1 reset not still connected to 0x56021c5601a0 2023-11-24T08:58:57.410 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T08:58:57.405+0000 7f4b5a023640 -1 reset not still connected to 0x56021d498a90 2023-11-24T08:58:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.export 2023-11-24T08:58:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:58:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_43.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.export 2023-11-24T08:58:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_43.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_43.export 2023-11-24T08:58:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:58:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_44 2023-11-24T08:58:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_44 snap 2023-11-24T08:58:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:58:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:58:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:58:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-11-24T08:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-11-24T08:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:58:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:58:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:58:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:59:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-11-24T08:59:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:59:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:59:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:59:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-11-24T08:59:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:59:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:59:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster=cluster1 2023-11-24T08:59:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ localgrep -i instance=cluster1 2023-11-24T08:59:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: state 2023-11-24T08:59:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:59:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:59:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:59:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:59:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:59:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:59:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:59:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:59:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:59:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:59:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:59:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:59:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:59:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:59:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-11-24T08:59:00.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:59:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:59:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:59:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:59:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:59:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:59:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-11-24T08:59:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:59:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:59:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:59:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-11-24T08:59:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:59:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_44.status 2023-11-24T08:59:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_44.status 2023-11-24T08:59:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:59:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_44.status 2023-11-24T08:59:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_44.status 2023-11-24T08:59:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_44.status 2023-11-24T08:59:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:59:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-11-24T08:59:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-11-24T08:59:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 205 2023-11-24T08:59:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_44.status 2023-11-24T08:59:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_44.status 2023-11-24T08:59:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:59:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=205 2023-11-24T08:59:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:59:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=111 2023-11-24T08:59:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 111 2023-11-24T08:59:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:59:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:02.533 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=111 2023-11-24T08:59:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 111 2023-11-24T08:59:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:59:02.642 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-24T08:59:02.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-24T08:59:02.643 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-11-24T08:59:02.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T08:59:02.642+0000 7f8e1d7df640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 49783) UID: 1000 2023-11-24T08:59:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 2023-11-24T08:59:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T08:59:04.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.646+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab336c80 [3/25e23892-6380-4923-94d3-6e58c6e982f0] start: already running 2023-11-24T08:59:04.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.670+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab336f00 [3/647ef1ae-5a98-459b-b31b-d5aa0404c570] start: already running 2023-11-24T08:59:04.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.678+0000 7f349befb640 -1 rbd::mirror::ImageReplayer: 0x55fd96014c80 [3/5b977de2-7224-446c-8040-6e733cf6a84b] start: already running 2023-11-24T08:59:04.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.678+0000 7f349befb640 -1 rbd::mirror::ImageReplayer: 0x55fd95fd9400 [3/8b099305-4776-4fa1-8f3c-5e18bc48f849] start: already running 2023-11-24T08:59:04.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.678+0000 7f349befb640 -1 rbd::mirror::ImageReplayer: 0x55fd95fd8000 [3/ab341c38-aeae-40e6-8dbf-7e351d9676c7] start: already running 2023-11-24T08:59:04.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.678+0000 7f349befb640 -1 rbd::mirror::ImageReplayer: 0x55fd96015b80 [3/16bfa33f-6709-47f2-9afa-1242602344ab] start: already running 2023-11-24T08:59:04.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.678+0000 7f349befb640 -1 rbd::mirror::ImageReplayer: 0x55fd96015180 [3/31318b39-6acb-4741-8297-a101c3fafabc] start: already running 2023-11-24T08:59:04.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.678+0000 7f349befb640 -1 rbd::mirror::ImageReplayer: 0x55fd96014f00 [3/5023e532-77d0-4a81-9a30-bf38f68789b1] start: already running 2023-11-24T08:59:04.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab336500 [3/3b2fce77-b38d-48ea-b5c5-cb78f30c2412] start: already running 2023-11-24T08:59:04.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab337b80 [3/9114b26e-4897-4df7-936d-4ec836b81477] start: already running 2023-11-24T08:59:04.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab336780 [3/7bc813da-10f2-42ef-b0da-2017c7edc945] start: already running 2023-11-24T08:59:04.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab336280 [3/170b38bb-5513-4ade-9ac7-4a9823217a98] start: already running 2023-11-24T08:59:04.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab337400 [3/cb1bf2ef-bc24-41f4-86ea-0b0c7b483a22] start: already running 2023-11-24T08:59:04.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab337180 [3/6956a2e8-c6e1-4b1f-8188-481ec5f6a81b] start: already running 2023-11-24T08:59:04.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab337900 [3/6e18b899-53e6-40ab-b047-1c57f3a06280] start: already running 2023-11-24T08:59:04.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cac234000 [3/c2dd7404-8a4d-4ca3-966a-b940cfe221e9] start: already running 2023-11-24T08:59:04.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab337680 [3/e79a8072-8f71-4674-9961-be59755f749a] start: already running 2023-11-24T08:59:04.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T08:59:04.686+0000 7fec7f5d8640 -1 rbd::mirror::ImageReplayer: 0x561cab336a00 [3/3cc6e0f9-790b-43bf-a15d-2b4b68e157bd] start: already running 2023-11-24T08:59:04.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.690+0000 7f349b6fa640 -1 rbd::mirror::ImageReplayer: 0x55fd96014780 [3/93be2a75-cd83-40a7-a3da-37d47064a00c] start: already running 2023-11-24T08:59:04.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.690+0000 7f349b6fa640 -1 rbd::mirror::ImageReplayer: 0x55fd96015900 [3/934c4e48-8c50-49b0-b632-f50d1f739feb] start: already running 2023-11-24T08:59:04.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.690+0000 7f349b6fa640 -1 rbd::mirror::ImageReplayer: 0x55fd95fd8280 [3/dd54726b-38b5-4534-a5b9-b85acdcdf924] start: already running 2023-11-24T08:59:04.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.690+0000 7f349b6fa640 -1 rbd::mirror::ImageReplayer: 0x55fd95fd9180 [3/f792a31d-dcf2-4828-bc5f-9490ec637a03] start: already running 2023-11-24T08:59:04.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.690+0000 7f349b6fa640 -1 rbd::mirror::ImageReplayer: 0x55fd96015400 [3/7c3b3b3c-f77d-402e-adec-8115e36a2f88] start: already running 2023-11-24T08:59:04.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T08:59:04.690+0000 7f349b6fa640 -1 rbd::mirror::ImageReplayer: 0x55fd96014a00 [3/e07e8277-39ef-4a01-8ed4-6a47771afe3f] start: already running 2023-11-24T08:59:04.884 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T08:59:04.876+0000 7fe3316f4640 -1 reset not still connected to 0x560363fa5520 2023-11-24T08:59:04.884 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T08:59:04.876+0000 7fe3316f4640 -1 reset not still connected to 0x560363fae8f0 2023-11-24T08:59:04.884 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T08:59:04.876+0000 7fe3316f4640 -1 reset not still connected to 0x560366548dd0 2023-11-24T08:59:04.884 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T08:59:04.876+0000 7fe3316f4640 -1 reset not still connected to 0x5603665492b0 2023-11-24T08:59:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 2023-11-24T08:59:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:59:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 2023-11-24T08:59:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T08:59:07.643 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-24T08:59:07.643 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T08:59:07.644 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-24T08:59:07.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-24T08:59:07.644 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-24T08:59:07.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-24T08:59:07.647 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-11-24T08:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 2023-11-24T08:59:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T08:59:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 2023-11-24T08:59:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T08:59:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 2023-11-24T08:59:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:59:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 2023-11-24T08:59:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T08:59:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 2023-11-24T08:59:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 16 2023-11-24T08:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.status 2023-11-24T08:59:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-24T08:59:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=205 2023-11-24T08:59:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 205 = 205 2023-11-24T08:59:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-11-24T08:59:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:59:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:59:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:59:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-11-24T08:59:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_44': 2023-11-24T08:59:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:59:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:59:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:59:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12a052bc9105 2023-11-24T08:59:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12a052bc9105 2023-11-24T08:59:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:59:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:59:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:59:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:59:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:36 2023 2023-11-24T08:59:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:36 2023 2023-11-24T08:59:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:36 2023 2023-11-24T08:59:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:59:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:59:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:59:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 6f744513-92ac-4212-8ca4-f89cc0e36fcf 2023-11-24T08:59:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:59:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_44 snap 2023-11-24T08:59:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T08:59:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:59:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:59:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_44.export 2023-11-24T08:59:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.export 2023-11-24T08:59:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_44.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.export 2023-11-24T08:59:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_44.export 2023-11-24T08:59:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:59:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.export 2023-11-24T08:59:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:59:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_44.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.export 2023-11-24T08:59:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_44.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_44.export 2023-11-24T08:59:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_45 2023-11-24T08:59:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_45 snap 2023-11-24T08:59:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:59:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T08:59:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-11-24T08:59:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-11-24T08:59:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T08:59:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:59:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:59:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-11-24T08:59:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T08:59:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T08:59:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T08:59:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_45 2023-11-24T08:59:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T08:59:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:59:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T08:59:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T08:59:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T08:59:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i state 2023-11-24T08:59:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T08:59:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T08:59:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-11-24T08:59:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T08:59:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T08:59:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:59:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T08:59:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:59:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T08:59:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T08:59:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T08:59:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T08:59:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:59:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:59:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T08:59:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-11-24T08:59:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T08:59:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T08:59:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T08:59:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T08:59:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T08:59:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T08:59:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-11-24T08:59:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:59:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:54.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T08:59:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T08:59:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-11-24T08:59:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T08:59:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_45.status 2023-11-24T08:59:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sedecho -e s|[/ ]|_|g cluster2-mirror-image_45.status 2023-11-24T08:59:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_45.status 2023-11-24T08:59:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_45.status 2023-11-24T08:59:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_45.status 2023-11-24T08:59:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T08:59:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.status 2023-11-24T08:59:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-11-24T08:59:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-11-24T08:59:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 207 2023-11-24T08:59:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_45.status 2023-11-24T08:59:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_45.status 2023-11-24T08:59:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-11-24T08:59:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T08:59:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=207 2023-11-24T08:59:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T08:59:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.status 2023-11-24T08:59:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.status 2023-11-24T08:59:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-11-24T08:59:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 207 = 2023-11-24T08:59:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T08:59:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.status 2023-11-24T08:59:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.status 2023-11-24T08:59:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-11-24T08:59:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T08:59:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T08:59:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T08:59:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T08:59:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T08:59:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 207 = 2023-11-24T08:59:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T08:59:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.status 2023-11-24T08:59:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.status 2023-11-24T08:59:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-11-24T08:59:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T08:59:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=207 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 207 = 207 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T08:59:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T08:59:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-11-24T08:59:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_45': 2023-11-24T08:59:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T08:59:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T08:59:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T08:59:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12b0c6e1e89f 2023-11-24T08:59:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12b0c6e1e89f 2023-11-24T08:59:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T08:59:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T08:59:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T08:59:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T08:59:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:49:46 2023 2023-11-24T08:59:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:49:46 2023 2023-11-24T08:59:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:49:46 2023 2023-11-24T08:59:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T08:59:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T08:59:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T08:59:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 5b977de2-7224-446c-8040-6e733cf6a84b 2023-11-24T08:59:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T08:59:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_45 snap 2023-11-24T08:59:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T08:59:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:59:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T08:59:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_45.export 2023-11-24T08:59:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.export 2023-11-24T08:59:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_45.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.export 2023-11-24T08:59:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_45.export 2023-11-24T08:59:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:59:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.export 2023-11-24T08:59:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T08:59:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_45.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.export 2023-11-24T08:59:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_45.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_45.export 2023-11-24T08:59:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T08:59:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_46 2023-11-24T08:59:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_46 snap 2023-11-24T08:59:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T08:59:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T08:59:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T08:59:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T08:59:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T09:00:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-11-24T09:00:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-11-24T09:00:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T09:00:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-11-24T09:00:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T09:00:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T09:00:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T09:00:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-11-24T09:00:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T09:00:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + local cluster=cluster1 2023-11-24T09:00:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T09:00:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instancegrep 2023-11-24T09:00:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -i state 2023-11-24T09:00:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T09:00:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T09:00:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-11-24T09:00:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-11-24T09:00:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T09:00:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T09:00:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2023-11-24T09:00:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T09:00:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T09:00:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T09:00:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T09:00:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T09:00:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T09:00:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T09:00:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T09:00:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2023-11-24T09:00:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T09:00:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T09:00:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T09:00:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T09:00:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-11-24T09:00:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_46.status 2023-11-24T09:00:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_46.status 2023-11-24T09:00:02.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T09:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_46.status 2023-11-24T09:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_46.status 2023-11-24T09:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_46.status 2023-11-24T09:00:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T09:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.status 2023-11-24T09:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-11-24T09:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-11-24T09:00:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 209 2023-11-24T09:00:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_46.status 2023-11-24T09:00:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_46.status 2023-11-24T09:00:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-11-24T09:00:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T09:00:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=209 2023-11-24T09:00:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T09:00:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.status 2023-11-24T09:00:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.status 2023-11-24T09:00:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-11-24T09:00:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=113 2023-11-24T09:00:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 209 = 113 2023-11-24T09:00:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T09:00:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.status 2023-11-24T09:00:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.status 2023-11-24T09:00:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-11-24T09:00:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 209 = 2023-11-24T09:00:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T09:00:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.status 2023-11-24T09:00:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.status 2023-11-24T09:00:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-11-24T09:00:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:04.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 209 = 2023-11-24T09:00:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T09:00:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.status 2023-11-24T09:00:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.status 2023-11-24T09:00:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-11-24T09:00:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=209 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 209 = 209 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-11-24T09:00:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_46': 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12b0e23d3cf5 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12b0e23d3cf5 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:50:01 2023 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:50:01 2023 2023-11-24T09:00:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:50:01 2023 2023-11-24T09:00:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T09:00:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:00:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:00:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 02bfc8e5-4c3a-4991-aa98-8ab3d6782ef1 2023-11-24T09:00:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T09:00:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_46 snap 2023-11-24T09:00:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:00:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T09:00:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_46.export 2023-11-24T09:00:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.export 2023-11-24T09:00:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_46.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.export 2023-11-24T09:00:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_46.export 2023-11-24T09:00:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.export 2023-11-24T09:00:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_46.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.export 2023-11-24T09:00:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_46.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_46.export 2023-11-24T09:00:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_47 2023-11-24T09:00:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_47 snap 2023-11-24T09:00:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:07.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T09:00:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-11-24T09:00:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-11-24T09:00:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T09:00:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-11-24T09:00:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T09:00:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T09:00:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T09:00:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_47 2023-11-24T09:00:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1+ 2023-11-24T09:00:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1grep 2023-11-24T09:00:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -i+ statelocal 2023-11-24T09:00:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: loop_instance 2023-11-24T09:00:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T09:00:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-11-24T09:00:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-11-24T09:00:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T09:00:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T09:00:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T09:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T09:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T09:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T09:00:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T09:00:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T09:00:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T09:00:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T09:00:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-11-24T09:00:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-11-24T09:00:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T09:00:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T09:00:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2023-11-24T09:00:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T09:00:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T09:00:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T09:00:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T09:00:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T09:00:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T09:00:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T09:00:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T09:00:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T09:00:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T09:00:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2023-11-24T09:00:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-11-24T09:00:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T09:00:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T09:00:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2023-11-24T09:00:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T09:00:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T09:00:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T09:00:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T09:00:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T09:00:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T09:00:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T09:00:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T09:00:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T09:00:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T09:00:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T09:00:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T09:00:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T09:00:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T09:00:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T09:00:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T09:00:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T09:00:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T09:00:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T09:00:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2023-11-24T09:00:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T09:00:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T09:00:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T09:00:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T09:00:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T09:00:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-11-24T09:00:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_47.status 2023-11-24T09:00:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_47.status 2023-11-24T09:00:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T09:00:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_47.status 2023-11-24T09:00:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_47.status 2023-11-24T09:00:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_47.status 2023-11-24T09:00:11.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T09:00:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.status 2023-11-24T09:00:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-11-24T09:00:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-11-24T09:00:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 211 2023-11-24T09:00:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_47.status 2023-11-24T09:00:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_47.status 2023-11-24T09:00:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-11-24T09:00:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T09:00:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=211 2023-11-24T09:00:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T09:00:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.status 2023-11-24T09:00:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.status 2023-11-24T09:00:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-11-24T09:00:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 211 = 2023-11-24T09:00:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T09:00:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.status 2023-11-24T09:00:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.status 2023-11-24T09:00:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-11-24T09:00:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 211 = 2023-11-24T09:00:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T09:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.status 2023-11-24T09:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.status 2023-11-24T09:00:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-11-24T09:00:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=211 2023-11-24T09:00:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 211 = 211 2023-11-24T09:00:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-11-24T09:00:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-11-24T09:00:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_47': 2023-11-24T09:00:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T09:00:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:00:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T09:00:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12a0899beb19 2023-11-24T09:00:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12a0899beb19 2023-11-24T09:00:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:00:15.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T09:00:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:00:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:00:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:50:12 2023 2023-11-24T09:00:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:50:12 2023 2023-11-24T09:00:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:50:12 2023 2023-11-24T09:00:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T09:00:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:00:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:00:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 5023e532-77d0-4a81-9a30-bf38f68789b1 2023-11-24T09:00:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T09:00:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_47 snap 2023-11-24T09:00:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:00:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T09:00:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_47.export 2023-11-24T09:00:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.export 2023-11-24T09:00:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_47.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.export 2023-11-24T09:00:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_47.export 2023-11-24T09:00:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.export 2023-11-24T09:00:15.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_47.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.export 2023-11-24T09:00:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_47.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_47.export 2023-11-24T09:00:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_48 2023-11-24T09:00:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_48 snap 2023-11-24T09:00:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T09:00:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-11-24T09:00:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T09:00:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T09:00:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T09:00:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-11-24T09:00:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local+ cluster_instance=cluster1 2023-11-24T09:00:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ greplocal -i state cluster=cluster1 2023-11-24T09:00:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T09:00:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T09:00:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T09:00:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T09:00:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-11-24T09:00:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T09:00:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T09:00:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T09:00:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T09:00:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-11-24T09:00:18.829 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6215510663443339) > weight (0.25) 2023-11-24T09:00:18.830 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-24T09:00:18.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-24T09:00:18.830 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-24T09:00:18.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:00:18.829+0000 7f6e5fe41640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 59638) UID: 1000 2023-11-24T09:00:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T09:00:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T09:00:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T09:00:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T09:00:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T09:00:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T09:00:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_48.status 2023-11-24T09:00:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sed -e s|[/ ]|_|g 2023-11-24T09:00:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:echo cluster2-mirror-image_48.status 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_48.status 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_48.status 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_48.status 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.status 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-11-24T09:00:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 213 2023-11-24T09:00:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_48.status 2023-11-24T09:00:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_48.status 2023-11-24T09:00:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-11-24T09:00:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T09:00:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=213 2023-11-24T09:00:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T09:00:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.status 2023-11-24T09:00:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.status 2023-11-24T09:00:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-11-24T09:00:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=115 2023-11-24T09:00:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 213 = 115 2023-11-24T09:00:20.146 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T09:00:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.status 2023-11-24T09:00:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.status 2023-11-24T09:00:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-11-24T09:00:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 213 = 2023-11-24T09:00:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T09:00:20.831 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-24T09:00:20.831 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:00:20.831 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-24T09:00:20.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-24T09:00:20.831 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-24T09:00:20.834 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-24T09:00:20.834 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2023-11-24T09:00:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.status 2023-11-24T09:00:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.status 2023-11-24T09:00:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-11-24T09:00:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 213 = 2023-11-24T09:00:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T09:00:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.status 2023-11-24T09:00:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.status 2023-11-24T09:00:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-11-24T09:00:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=213 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 213 = 213 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-11-24T09:00:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_48': 2023-11-24T09:00:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T09:00:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:00:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T09:00:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12b03a70d225 2023-11-24T09:00:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12b03a70d225 2023-11-24T09:00:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:00:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T09:00:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:00:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:00:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:50:23 2023 2023-11-24T09:00:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:50:23 2023 2023-11-24T09:00:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:50:23 2023 2023-11-24T09:00:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T09:00:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:00:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:00:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 934c4e48-8c50-49b0-b632-f50d1f739feb 2023-11-24T09:00:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T09:00:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_48 snap 2023-11-24T09:00:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:00:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T09:00:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_48.export 2023-11-24T09:00:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.export 2023-11-24T09:00:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_48.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.export 2023-11-24T09:00:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_48.export 2023-11-24T09:00:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.export 2023-11-24T09:00:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_48.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.export 2023-11-24T09:00:25.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_48.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_48.export 2023-11-24T09:00:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_49 2023-11-24T09:00:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_49 snap 2023-11-24T09:00:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T09:00:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-11-24T09:00:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_49+ 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grepshift 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -i state 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T09:00:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T09:00:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T09:00:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-11-24T09:00:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T09:00:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T09:00:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T09:00:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T09:00:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-11-24T09:00:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-11-24T09:00:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T09:00:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T09:00:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2023-11-24T09:00:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T09:00:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T09:00:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T09:00:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T09:00:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T09:00:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T09:00:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T09:00:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T09:00:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T09:00:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T09:00:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T09:00:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:27.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T09:00:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:27.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2023-11-24T09:00:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-11-24T09:00:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T09:00:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T09:00:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T09:00:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T09:00:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T09:00:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T09:00:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T09:00:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T09:00:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T09:00:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T09:00:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2023-11-24T09:00:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T09:00:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + echogrep "state": "Replaying" -i Replaying 2023-11-24T09:00:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T09:00:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T09:00:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T09:00:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-11-24T09:00:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T09:00:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T09:00:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-11-24T09:00:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_49.status 2023-11-24T09:00:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_49.status 2023-11-24T09:00:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T09:00:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_49.status 2023-11-24T09:00:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_49.status 2023-11-24T09:00:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_49.status 2023-11-24T09:00:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T09:00:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.status 2023-11-24T09:00:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-11-24T09:00:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-11-24T09:00:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 215 2023-11-24T09:00:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_49.status 2023-11-24T09:00:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_49.status 2023-11-24T09:00:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-11-24T09:00:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T09:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=215 2023-11-24T09:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T09:00:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.status 2023-11-24T09:00:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.status 2023-11-24T09:00:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-11-24T09:00:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=116 2023-11-24T09:00:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 215 = 116 2023-11-24T09:00:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T09:00:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.status 2023-11-24T09:00:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.status 2023-11-24T09:00:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-11-24T09:00:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:29.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 215 = 2023-11-24T09:00:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T09:00:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.status 2023-11-24T09:00:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.status 2023-11-24T09:00:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-11-24T09:00:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 215 = 2023-11-24T09:00:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T09:00:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.status 2023-11-24T09:00:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.status 2023-11-24T09:00:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-11-24T09:00:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=215 2023-11-24T09:00:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 215 = 215 2023-11-24T09:00:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-11-24T09:00:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-11-24T09:00:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_49': 2023-11-24T09:00:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T09:00:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:00:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T09:00:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12071a7dc2b0 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12071a7dc2b0 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:50:31 2023 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:50:31 2023 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:50:31 2023 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:00:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: c2dd7404-8a4d-4ca3-966a-b940cfe221e9 2023-11-24T09:00:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T09:00:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_49 snap 2023-11-24T09:00:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:00:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T09:00:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_49.export 2023-11-24T09:00:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.export 2023-11-24T09:00:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_49.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.export 2023-11-24T09:00:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_49.export 2023-11-24T09:00:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.export 2023-11-24T09:00:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_49.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.export 2023-11-24T09:00:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_49.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_49.export 2023-11-24T09:00:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_50 2023-11-24T09:00:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_snap cluster2 mirror image_50 snap 2023-11-24T09:00:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-11-24T09:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-11-24T09:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-11-24T09:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=started 2023-11-24T09:00:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:35.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:36.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-11-24T09:00:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=started 2023-11-24T09:00:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_result 2023-11-24T09:00:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=stopped 2023-11-24T09:00:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-11-24T09:00:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_instance=cluster1 2023-11-24T09:00:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift+ 2023-11-24T09:00:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance=cluster1 2023-11-24T09:00:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loop_instance 2023-11-24T09:00:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -i state 2023-11-24T09:00:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 != cluster1 ] 2023-11-24T09:00:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T09:00:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:0 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:0 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 = cluster1:0 ] 2023-11-24T09:00:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:00:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:00:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-11-24T09:00:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-24T09:00:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-11-24T09:00:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-11-24T09:00:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T09:00:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T09:00:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2023-11-24T09:00:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T09:00:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T09:00:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-11-24T09:00:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:1 2023-11-24T09:00:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:1 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 1 = cluster1:1 ] 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=1 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=1 2023-11-24T09:00:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-11-24T09:00:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-24T09:00:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2023-11-24T09:00:37.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-11-24T09:00:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T09:00:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T09:00:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2023-11-24T09:00:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T09:00:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-24T09:00:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T09:00:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:37.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:2 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-11-24T09:00:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:2 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 2 = cluster1:2 ] 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=2 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=2 2023-11-24T09:00:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-11-24T09:00:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T09:00:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T09:00:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-24T09:00:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2023-11-24T09:00:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-11-24T09:00:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd mirror status : get status for rbd mirror 2023-11-24T09:00:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:admin_socket: invalid command 2023-11-24T09:00:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2023-11-24T09:00:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T09:00:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:00:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-24T09:00:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T09:00:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:00:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:37.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T09:00:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T09:00:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:00:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T09:00:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T09:00:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-11-24T09:00:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1:3 2023-11-24T09:00:37.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:00:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-11-24T09:00:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1:3 2023-11-24T09:00:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=local_cluster 2023-11-24T09:00:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:00:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:00:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T09:00:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 3 = cluster1:3 ] 2023-11-24T09:00:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval local_cluster=cluster1 2023-11-24T09:00:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local_cluster=cluster1 2023-11-24T09:00:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=3 2023-11-24T09:00:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=3 2023-11-24T09:00:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-11-24T09:00:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T09:00:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T09:00:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-24T09:00:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2023-11-24T09:00:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ status_result= "state": "Replaying" 2023-11-24T09:00:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo "state": "Replaying" 2023-11-24T09:00:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -i Replaying 2023-11-24T09:00:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: "state": "Replaying" 2023-11-24T09:00:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=started 2023-11-24T09:00:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test started = started 2023-11-24T09:00:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-11-24T09:00:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = journal ] 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ snapshot = snapshot ] 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_cluster=cluster1 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster2-mirror-image_50.status 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster2-mirror-image_50.status+ 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sed -e s|[/ ]|_|g 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local status_log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_50.status 2023-11-24T09:00:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mkfname cluster1-mirror-image_50.status 2023-11-24T09:00:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo cluster1-mirror-image_50.status 2023-11-24T09:00:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -e s|[/ ]|_|g 2023-11-24T09:00:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local local_status_log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.status 2023-11-24T09:00:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-11-24T09:00:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-11-24T09:00:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Snapshot ID: 217 2023-11-24T09:00:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_50.status 2023-11-24T09:00:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_50.status 2023-11-24T09:00:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-11-24T09:00:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-11-24T09:00:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snapshot_id=217 2023-11-24T09:00:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:37.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.2 2023-11-24T09:00:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.status 2023-11-24T09:00:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.status 2023-11-24T09:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-11-24T09:00:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=117 2023-11-24T09:00:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 217 = 117 2023-11-24T09:00:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.4 2023-11-24T09:00:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.status 2023-11-24T09:00:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.status 2023-11-24T09:00:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-11-24T09:00:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 217 = 2023-11-24T09:00:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.8 2023-11-24T09:00:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.status 2023-11-24T09:00:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.status 2023-11-24T09:00:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-11-24T09:00:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:00:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:-:1.1: Document is empty 2023-11-24T09:00:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: 2023-11-24T09:00:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:^ 2023-11-24T09:00:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id= 2023-11-24T09:00:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 217 = 2023-11-24T09:00:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1.6 2023-11-24T09:00:39.532 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T09:00:39.529+0000 7fee2f6de640 -1 reset not still connected to 0x55a946f77ba0 2023-11-24T09:00:39.532 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T09:00:39.529+0000 7fee2f6de640 -1 reset not still connected to 0x55a94f5bfe10 2023-11-24T09:00:39.534 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T09:00:39.533+0000 7fee2f6de640 -1 reset not still connected to 0x55a95a95bad0 2023-11-24T09:00:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.status 2023-11-24T09:00:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local log=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.status 2023-11-24T09:00:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-11-24T09:00:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-11-24T09:00:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-24T09:00:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local primary_snapshot_id=217 2023-11-24T09:00:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 217 = 217 2023-11-24T09:00:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-11-24T09:00:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:00:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:00:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:00:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-11-24T09:00:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_50': 2023-11-24T09:00:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 128 MiB in 32 objects 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 2 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 12876950d3f0 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.12876950d3f0 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 08:50:43 2023 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 08:50:43 2023 2023-11-24T09:00:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 08:50:43 2023 2023-11-24T09:00:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: protected: False 2023-11-24T09:00:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:00:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:00:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 16bfa33f-6709-47f2-9afa-1242602344ab 2023-11-24T09:00:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: false 2023-11-24T09:00:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ compare_image_snaps mirror image_50 snap 2023-11-24T09:00:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:00:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap_name=snap 2023-11-24T09:00:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local ret=0 2023-11-24T09:00:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local rmt_export=/tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_50.export 2023-11-24T09:00:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local loc_export=/tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.export 2023-11-24T09:00:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_50.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.export 2023-11-24T09:00:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_50.export 2023-11-24T09:00:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.export 2023-11-24T09:00:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-24T09:00:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cmp /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_50.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.export 2023-11-24T09:00:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -f /tmp/tmp.tyy3Nh9UuT/cluster2-mirror-image_50.export /tmp/tmp.tyy3Nh9UuT/cluster1-mirror-image_50.export 2023-11-24T09:00:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog TEST: delete many images 2023-11-24T09:00:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a+ /tmp/tmp.tyy3Nh9UuT/rbd-mirror.test.log 2023-11-24T09:00:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:date +%F %T 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2023-11-24 09:00:43 TEST: delete many images 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2023-11-24 09:00:43 TEST: delete many images 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 50 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_1 2023-11-24T09:00:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:00:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_1 2023-11-24T09:00:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:00:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-11-24T09:00:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-11-24T09:00:46.047 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:00:44.101+0000 7f2718310640 -1 rbd::mirror::ImageMap: 0x557e4fdf1320 handle_peer_ack_remove: failed to remove global_image_id=31318b39-6acb-4741-8297-a101c3fafabc 2023-11-24T09:00:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:00:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2023-11-24T09:00:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-11-24T09:00:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:00:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:00:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-11-24T09:00:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:00:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_2 2023-11-24T09:00:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:00:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_2 2023-11-24T09:00:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:00:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-11-24T09:00:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-11-24T09:00:47.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:00:47.085+0000 7f2711b03640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e5017e000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:00:47.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:00:47.085+0000 7f2718b11640 -1 rbd::mirror::ImageReplayer: 0x557e4f94f400 [3/f38cb77f-9d83-40c5-a0c0-ff19cc94a15c] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:00:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:00:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2023-11-24T09:00:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-11-24T09:00:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:00:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:00:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-11-24T09:00:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_3 2023-11-24T09:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_3 2023-11-24T09:00:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:00:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-11-24T09:00:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-11-24T09:00:50.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:00:50.105+0000 7f298c47e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a5b036c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:00:50.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:00:50.105+0000 7f2992c8b640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce2500 [3/0910fa58-c0e2-4ebc-ba44-f7fe28530964] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:00:51.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:00:51.001+0000 7f298bc7d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a59e8000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:00:51.003 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:00:51.001+0000 7f299248a640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce2500 [3/0910fa58-c0e2-4ebc-ba44-f7fe28530964] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:00:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:00:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2023-11-24T09:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-11-24T09:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:00:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-11-24T09:00:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:00:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_4 2023-11-24T09:00:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:00:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_4 2023-11-24T09:00:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:00:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-11-24T09:00:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-11-24T09:00:53.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:00:53.153+0000 7f2712304640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e5017e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:00:53.157 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:00:53.153+0000 7f2718b11640 -1 rbd::mirror::ImageReplayer: 0x557e4f94ec80 [3/eb96c510-4f18-49d6-8bef-2744c8373fbf] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:00:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:00:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2023-11-24T09:00:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-11-24T09:00:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:00:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:00:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-11-24T09:00:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:00:55.485 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T09:00:55.484+0000 7fee2f6de640 -1 reset not still connected to 0x55a9563b89c0 2023-11-24T09:00:55.487 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T09:00:55.484+0000 7fee2f6de640 -1 reset not still connected to 0x55a9563b9d40 2023-11-24T09:00:55.488 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T09:00:55.484+0000 7fee2f6de640 -1 reset not still connected to 0x55a95923a0d0 2023-11-24T09:00:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_5 2023-11-24T09:00:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:00:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_5 2023-11-24T09:00:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:00:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-11-24T09:00:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-11-24T09:00:55.801 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:00:55.800+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53dc4340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:00:55.802 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:00:55.800+0000 7fb5db1ad640 -1 rbd::mirror::ImageReplayer: 0x558a52f3e780 [3/14adee4f-4c4d-4e0f-8207-88b9d7b5b7af] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:00:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:00:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:00:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2023-11-24T09:00:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-11-24T09:00:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:00:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:00:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-11-24T09:00:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:00:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_6 2023-11-24T09:00:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:00:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:00:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_6 2023-11-24T09:00:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:00:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:00:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:00:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-11-24T09:00:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-11-24T09:00:58.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:00:58.844+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959accb1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:00:58.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:00:58.844+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959ab5aa00 [3/f792a31d-dcf2-4828-bc5f-9490ec637a03] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:00.770 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2023-11-24T09:01:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-11-24T09:01:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:01:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-11-24T09:01:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_7 2023-11-24T09:01:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:01:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_7 2023-11-24T09:01:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:01:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-11-24T09:01:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-11-24T09:01:01.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:01.856+0000 7f2711b03640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e504af520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:01.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:01.856+0000 7f2718b11640 -1 rbd::mirror::ImageReplayer: 0x557e4f94f680 [3/e07e8277-39ef-4a01-8ed4-6a47771afe3f] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2023-11-24T09:01:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-11-24T09:01:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:01:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-11-24T09:01:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_8 2023-11-24T09:01:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:01:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_8 2023-11-24T09:01:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:01:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-11-24T09:01:04.647 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.47485506346703854) > weight (0.25) 2023-11-24T09:01:04.648 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9773179414150505) > weight (0.25) 2023-11-24T09:01:04.648 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-11-24T09:01:04.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-11-24T09:01:04.648 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-11-24T09:01:04.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:01:04.648+0000 7fec8cdf3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 53479) UID: 1000 2023-11-24T09:01:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-11-24T09:01:04.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:04.868+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959accb380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:04.869 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:04.868+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959b00bb80 [3/8b099305-4776-4fa1-8f3c-5e18bc48f849] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2023-11-24T09:01:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-11-24T09:01:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:01:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-11-24T09:01:07.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_9 2023-11-24T09:01:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:01:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_9 2023-11-24T09:01:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:01:07.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-11-24T09:01:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-11-24T09:01:07.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:07.812+0000 7f298c47e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a5669ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:07.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:07.812+0000 7f299248a640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce3680 [3/18de629c-76f5-4bac-9f50-e3a0a137e23c] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:09.649 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-11-24T09:01:09.649 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:01:09.650 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-11-24T09:01:09.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-11-24T09:01:09.650 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-11-24T09:01:09.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-11-24T09:01:09.652 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-11-24T09:01:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2023-11-24T09:01:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-11-24T09:01:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:01:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-11-24T09:01:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_10 2023-11-24T09:01:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:01:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_10 2023-11-24T09:01:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:01:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-11-24T09:01:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-11-24T09:01:11.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:11.056+0000 7f024411a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959acca9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:11.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:11.056+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959b00af00 [3/342ef9c2-57fd-4085-be72-fae4ddc7c8a0] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_11 2023-11-24T09:01:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-11-24T09:01:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T09:01:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-11-24T09:01:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_11 2023-11-24T09:01:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T09:01:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_11 2023-11-24T09:01:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T09:01:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-11-24T09:01:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-11-24T09:01:13.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:13.868+0000 7f298bc7d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a5669d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:13.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:13.868+0000 7f299248a640 -1 rbd::mirror::ImageReplayer: 0x5640a4cd6000 [3/93be2a75-cd83-40a7-a3da-37d47064a00c] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:15.662 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:15.658+0000 7fe3316f4640 -1 reset not still connected to 0x560363fae8f0 2023-11-24T09:01:15.662 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:15.658+0000 7fe3316f4640 -1 reset not still connected to 0x560364150c30 2023-11-24T09:01:15.702 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:15.702+0000 7fe3316f4640 -1 reset not still connected to 0x560366548dd0 2023-11-24T09:01:15.721 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:15.718+0000 7fe3316f4640 -1 reset not still connected to 0x5603665492b0 2023-11-24T09:01:15.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_12 2023-11-24T09:01:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-11-24T09:01:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T09:01:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-11-24T09:01:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_12 2023-11-24T09:01:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T09:01:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_12 2023-11-24T09:01:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T09:01:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-11-24T09:01:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-11-24T09:01:16.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:16.892+0000 7f298c47e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a5b03520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:16.895 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:16.892+0000 7f299248a640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce2280 [3/08429c44-38df-4fe3-933e-3a53c6f058ba] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_13 2023-11-24T09:01:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-11-24T09:01:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T09:01:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-11-24T09:01:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_13 2023-11-24T09:01:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T09:01:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_13 2023-11-24T09:01:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T09:01:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-11-24T09:01:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-11-24T09:01:19.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:19.896+0000 7f2711b03640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e50a2d380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:19.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:19.896+0000 7f2718310640 -1 rbd::mirror::ImageReplayer: 0x557e4f94f900 [3/25e23892-6380-4923-94d3-6e58c6e982f0] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_14 2023-11-24T09:01:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-11-24T09:01:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T09:01:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-11-24T09:01:22.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_14 2023-11-24T09:01:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T09:01:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_14 2023-11-24T09:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T09:01:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-11-24T09:01:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-11-24T09:01:22.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:22.936+0000 7f2711b03640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e504ae820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:22.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:22.936+0000 7f2718b11640 -1 rbd::mirror::ImageReplayer: 0x557e50a32000 [3/2cfb09ab-c214-4e47-89a7-d051e9682117] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_15 2023-11-24T09:01:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-11-24T09:01:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T09:01:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-11-24T09:01:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_15 2023-11-24T09:01:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T09:01:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_15 2023-11-24T09:01:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T09:01:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-11-24T09:01:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-11-24T09:01:26.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:26.080+0000 7f2712304640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e50a2d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:26.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:26.080+0000 7f2718310640 -1 rbd::mirror::ImageReplayer: 0x557e4f94e780 [3/f1cebaf0-aa03-4dea-9666-f49bdf41a7b2] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_16 2023-11-24T09:01:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-11-24T09:01:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T09:01:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-11-24T09:01:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:28.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_16 2023-11-24T09:01:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T09:01:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_16 2023-11-24T09:01:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T09:01:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-11-24T09:01:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-11-24T09:01:29.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:29.000+0000 7f298bc7d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a579b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:29.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:29.000+0000 7f299248a640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce3400 [3/00787dee-37ed-4352-8685-7ea4d59a610b] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_17 2023-11-24T09:01:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-11-24T09:01:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T09:01:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-11-24T09:01:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_17 2023-11-24T09:01:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T09:01:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_17 2023-11-24T09:01:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T09:01:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-11-24T09:01:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-11-24T09:01:32.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:32.068+0000 7f298c47e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a579a340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:32.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:32.068+0000 7f2992c8b640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce2780 [3/12e91ddf-7a62-43f6-8172-24011e6b5d43] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_18 2023-11-24T09:01:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-11-24T09:01:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T09:01:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-11-24T09:01:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_18 2023-11-24T09:01:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T09:01:34.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_18 2023-11-24T09:01:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T09:01:34.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-11-24T09:01:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-11-24T09:01:35.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:35.084+0000 7f2712304640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e50a2c000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:35.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:35.084+0000 7f2718310640 -1 rbd::mirror::ImageReplayer: 0x557e50a32280 [3/de67f348-91f1-4254-a508-55ce6ec8848d] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_19 2023-11-24T09:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-11-24T09:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T09:01:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-11-24T09:01:36.799 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.797+0000 7fe3316f4640 -1 reset not still connected to 0x560361a14680 2023-11-24T09:01:36.799 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.797+0000 7fe3316f4640 -1 reset not still connected to 0x5603623a1c70 2023-11-24T09:01:36.800 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.797+0000 7fe3316f4640 -1 reset not still connected to 0x560363fa5520 2023-11-24T09:01:36.810 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.805+0000 7fe3316f4640 -1 reset not still connected to 0x560363fae8f0 2023-11-24T09:01:36.811 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.805+0000 7fe3316f4640 -1 reset not still connected to 0x560364150c30 2023-11-24T09:01:36.811 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.805+0000 7fe3316f4640 -1 reset not still connected to 0x560366548dd0 2023-11-24T09:01:36.811 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.805+0000 7fe3316f4640 -1 reset not still connected to 0x5603665492b0 2023-11-24T09:01:36.919 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.917+0000 7fe3316f4640 -1 reset not still connected to 0x5603623a1c70 2023-11-24T09:01:36.919 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.917+0000 7fe3316f4640 -1 reset not still connected to 0x560363fa5520 2023-11-24T09:01:36.919 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.917+0000 7fe3316f4640 -1 reset not still connected to 0x560363fae8f0 2023-11-24T09:01:36.919 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.917+0000 7fe3316f4640 -1 reset not still connected to 0x560364150c30 2023-11-24T09:01:36.919 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.917+0000 7fe3316f4640 -1 reset not still connected to 0x560366548dd0 2023-11-24T09:01:36.929 INFO:tasks.ceph.cluster1.osd.0.smithi153.stderr:2023-11-24T09:01:36.925+0000 7fe3316f4640 -1 reset not still connected to 0x5603665492b0 2023-11-24T09:01:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_19 2023-11-24T09:01:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T09:01:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_19 2023-11-24T09:01:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T09:01:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-11-24T09:01:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-11-24T09:01:38.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:01:38.936+0000 7fb5d49a0640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53f37040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:38.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:01:38.936+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a52f3e500 [3/7bc813da-10f2-42ef-b0da-2017c7edc945] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_20 2023-11-24T09:01:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-11-24T09:01:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T09:01:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-11-24T09:01:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:40.523 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_20 2023-11-24T09:01:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T09:01:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_20 2023-11-24T09:01:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T09:01:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-11-24T09:01:40.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-11-24T09:01:40.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:40.904+0000 7f2712304640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e504af6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:40.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:01:40.904+0000 7f2718b11640 -1 rbd::mirror::ImageReplayer: 0x557e4f94e500 [3/9f9af0b3-95c5-474e-831d-a21ecd5dc147] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_21 2023-11-24T09:01:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-11-24T09:01:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T09:01:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-11-24T09:01:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_21 2023-11-24T09:01:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T09:01:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_21 2023-11-24T09:01:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T09:01:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-11-24T09:01:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-11-24T09:01:43.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:01:43.888+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53f371e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:43.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:01:43.888+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a52f3ec80 [3/3cc6e0f9-790b-43bf-a15d-2b4b68e157bd] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_22 2023-11-24T09:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-11-24T09:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T09:01:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-11-24T09:01:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_22 2023-11-24T09:01:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T09:01:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_22 2023-11-24T09:01:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T09:01:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-11-24T09:01:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-11-24T09:01:46.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:46.899+0000 7f298bc7d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a579a680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:46.904 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:46.899+0000 7f2992c8b640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce2c80 [3/7c3b3b3c-f77d-402e-adec-8115e36a2f88] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_23 2023-11-24T09:01:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-11-24T09:01:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T09:01:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-11-24T09:01:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_23 2023-11-24T09:01:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T09:01:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_23 2023-11-24T09:01:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T09:01:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-11-24T09:01:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-11-24T09:01:49.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:49.923+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad949c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:49.927 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:49.923+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959b00b400 [3/6e18b899-53e6-40ab-b047-1c57f3a06280] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:49.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:49.991+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad95040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:49.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:49.991+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959b00b400 [3/6e18b899-53e6-40ab-b047-1c57f3a06280] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_24 2023-11-24T09:01:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-11-24T09:01:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T09:01:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-11-24T09:01:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_24 2023-11-24T09:01:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T09:01:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_24 2023-11-24T09:01:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T09:01:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-11-24T09:01:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-11-24T09:01:52.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:52.947+0000 7f298c47e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a5b03d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:52.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:01:52.947+0000 7f2992c8b640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce3900 [3/2238b317-dd07-4472-9062-bd252e8143ec] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_25 2023-11-24T09:01:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-11-24T09:01:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T09:01:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-11-24T09:01:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_25 2023-11-24T09:01:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T09:01:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_25 2023-11-24T09:01:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T09:01:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-11-24T09:01:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-11-24T09:01:55.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:55.863+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad951e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:55.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:55.863+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959b00a500 [3/9114b26e-4897-4df7-936d-4ec836b81477] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:01:57.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:01:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:01:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_26 2023-11-24T09:01:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-11-24T09:01:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T09:01:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:01:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-11-24T09:01:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:01:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_26 2023-11-24T09:01:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T09:01:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:01:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_26 2023-11-24T09:01:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:01:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:01:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T09:01:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-11-24T09:01:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-11-24T09:01:58.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:58.983+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad95520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:01:58.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:01:58.983+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959b00b900 [3/e79a8072-8f71-4674-9961-be59755f749a] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_27 2023-11-24T09:02:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-11-24T09:02:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T09:02:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-11-24T09:02:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_27 2023-11-24T09:02:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T09:02:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_27 2023-11-24T09:02:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T09:02:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-11-24T09:02:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-11-24T09:02:02.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:02.055+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53d056c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:02.060 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:02.055+0000 7fb5db1ad640 -1 rbd::mirror::ImageReplayer: 0x558a52f3ea00 [3/170b38bb-5513-4ade-9ac7-4a9823217a98] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_28 2023-11-24T09:02:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-11-24T09:02:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T09:02:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:03.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-11-24T09:02:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:04.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_28 2023-11-24T09:02:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T09:02:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_28 2023-11-24T09:02:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T09:02:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-11-24T09:02:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-11-24T09:02:05.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:05.091+0000 7f2712304640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e504aeb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:05.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:05.091+0000 7f2718b11640 -1 rbd::mirror::ImageReplayer: 0x557e50a32a00 [3/dd54726b-38b5-4534-a5b9-b85acdcdf924] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_29 2023-11-24T09:02:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-11-24T09:02:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T09:02:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:06.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-11-24T09:02:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_29 2023-11-24T09:02:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T09:02:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_29 2023-11-24T09:02:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T09:02:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-11-24T09:02:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-11-24T09:02:07.816 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T09:02:07.813+0000 7f4b5a023640 -1 reset not still connected to 0x560217eccf70 2023-11-24T09:02:07.816 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T09:02:07.813+0000 7f4b5a023640 -1 reset not still connected to 0x5602195f1e10 2023-11-24T09:02:07.817 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T09:02:07.813+0000 7f4b5a023640 -1 reset not still connected to 0x56021eacc410 2023-11-24T09:02:07.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:07.871+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad944e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:07.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:07.871+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959b00a780 [3/9ed4d512-1473-4b48-9cd4-c90c2f60808e] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_30 2023-11-24T09:02:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-11-24T09:02:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T09:02:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:09.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-11-24T09:02:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_30 2023-11-24T09:02:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T09:02:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_30 2023-11-24T09:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T09:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-11-24T09:02:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-11-24T09:02:10.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:10.891+0000 7f2712304640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557e50998b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:10.894 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:10.891+0000 7f2718310640 -1 rbd::mirror::ImageReplayer: 0x557e4f94ef00 [3/d77de178-524d-4cb8-ba9c-8399123b0719] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_31 2023-11-24T09:02:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-11-24T09:02:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T09:02:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-11-24T09:02:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_31 2023-11-24T09:02:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T09:02:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_31 2023-11-24T09:02:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T09:02:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-11-24T09:02:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-11-24T09:02:13.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:13.955+0000 7f024411a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad94b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:13.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:13.955+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959b00b680 [3/736a6ea1-a614-4fbc-8e6f-f5a28a8d570f] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:14.835 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-11-24T09:02:14.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-11-24T09:02:14.836 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-11-24T09:02:14.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:14.831+0000 7f2719312640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 44648) UID: 1000 2023-11-24T09:02:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_32 2023-11-24T09:02:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-11-24T09:02:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T09:02:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-11-24T09:02:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_32 2023-11-24T09:02:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T09:02:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_32 2023-11-24T09:02:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T09:02:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-11-24T09:02:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-11-24T09:02:16.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:16.927+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53f37520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:16.930 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:16.927+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a5391a500 [3/fea5c78b-14c3-43da-93a2-d38278c1541a] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:17.779 INFO:tasks.ceph.cluster1.osd.1.smithi153.stderr:2023-11-24T09:02:17.773+0000 7f9978959640 -1 reset not still connected to 0x55df08e348f0 2023-11-24T09:02:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_33 2023-11-24T09:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-11-24T09:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T09:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-11-24T09:02:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_33 2023-11-24T09:02:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T09:02:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_33 2023-11-24T09:02:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T09:02:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-11-24T09:02:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-11-24T09:02:19.836 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-11-24T09:02:19.837 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:02:19.837 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-11-24T09:02:19.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-11-24T09:02:19.838 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-11-24T09:02:19.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-11-24T09:02:19.841 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2023-11-24T09:02:20.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:20.003+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53f376c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:20.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:20.003+0000 7fb5db1ad640 -1 rbd::mirror::ImageReplayer: 0x558a52f3ef00 [3/6956a2e8-c6e1-4b1f-8188-481ec5f6a81b] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:20.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:20.783+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53f36b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:20.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:20.783+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a52f3ef00 [3/6956a2e8-c6e1-4b1f-8188-481ec5f6a81b] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_34 2023-11-24T09:02:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-11-24T09:02:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T09:02:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-11-24T09:02:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_34 2023-11-24T09:02:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T09:02:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_34 2023-11-24T09:02:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T09:02:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-11-24T09:02:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-11-24T09:02:23.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:23.019+0000 7fb5d49a0640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53f0b6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:23.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:23.019+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a52f3f680 [3/3d5be2fc-e83f-4e85-8d39-4fe13e555474] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_35 2023-11-24T09:02:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-11-24T09:02:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T09:02:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-11-24T09:02:25.538 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_35 2023-11-24T09:02:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T09:02:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_35 2023-11-24T09:02:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T09:02:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-11-24T09:02:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-11-24T09:02:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:02:25.875+0000 7f298bc7d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a57e9ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:25.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:02:25.875+0000 7f2992c8b640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce2a00 [3/c8458320-3838-4775-b00c-eeaf0f63bb66] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_36 2023-11-24T09:02:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-11-24T09:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T09:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-11-24T09:02:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_36 2023-11-24T09:02:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T09:02:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_36 2023-11-24T09:02:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T09:02:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-11-24T09:02:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-11-24T09:02:29.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:02:29.003+0000 7f349befb640 -1 rbd::mirror::ImageReplayer: 0x55fd96014c80 [3/fc86b320-d291-4bfa-a035-067b30e073fa] start: already running 2023-11-24T09:02:29.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:02:29.003+0000 7f349befb640 -1 rbd::mirror::ImageReplayer: 0x55fd96014c80 [3/fc86b320-d291-4bfa-a035-067b30e073fa] start: already running 2023-11-24T09:02:29.070 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:29.067+0000 7fb5da9ac640 -1 rbd::mirror::ImageMap: 0x558a533767e0 handle_peer_ack_remove: failed to remove global_image_id=fc86b320-d291-4bfa-a035-067b30e073fa 2023-11-24T09:02:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_37 2023-11-24T09:02:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-11-24T09:02:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T09:02:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-11-24T09:02:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_37 2023-11-24T09:02:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T09:02:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_37 2023-11-24T09:02:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T09:02:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-11-24T09:02:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-11-24T09:02:32.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:02:32.075+0000 7f298c47e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a57e9a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:32.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:02:32.075+0000 7f2992c8b640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce3180 [3/cb1bf2ef-bc24-41f4-86ea-0b0c7b483a22] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_38 2023-11-24T09:02:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-11-24T09:02:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T09:02:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-11-24T09:02:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_38 2023-11-24T09:02:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T09:02:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_38 2023-11-24T09:02:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T09:02:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-11-24T09:02:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-11-24T09:02:35.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:35.143+0000 7f549247e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a5d4d2000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:35.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:35.143+0000 7f549848a640 -1 rbd::mirror::ImageReplayer: 0x558a5cbe2500 [3/3b2fce77-b38d-48ea-b5c5-cb78f30c2412] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_39 2023-11-24T09:02:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-11-24T09:02:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T09:02:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-11-24T09:02:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_39 2023-11-24T09:02:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T09:02:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_39 2023-11-24T09:02:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T09:02:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-11-24T09:02:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-11-24T09:02:37.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:37.887+0000 7f549247e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a5d4d3040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:37.889 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:37.887+0000 7f5498c8b640 -1 rbd::mirror::ImageReplayer: 0x558a5cbe2280 [3/344f7f3d-a4fa-452d-94e2-e36d500bd26a] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_40 2023-11-24T09:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-11-24T09:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T09:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-11-24T09:02:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_40 2023-11-24T09:02:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T09:02:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_40 2023-11-24T09:02:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T09:02:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-11-24T09:02:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-11-24T09:02:40.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:40.907+0000 7f024411a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ae50000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:40.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:40.907+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959b00aa00 [3/b34477ff-894a-40fc-ba47-b95068f79b4c] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:41.669 INFO:tasks.ceph.cluster1.osd.1.smithi153.stderr:2023-11-24T09:02:41.664+0000 7f9978959640 -1 reset not still connected to 0x55df08e348f0 2023-11-24T09:02:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_41 2023-11-24T09:02:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-11-24T09:02:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T09:02:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-11-24T09:02:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_41 2023-11-24T09:02:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T09:02:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_41 2023-11-24T09:02:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T09:02:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-11-24T09:02:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-11-24T09:02:43.967 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:43.966+0000 7fb5da9ac640 -1 rbd::mirror::ImageMap: 0x558a533767e0 handle_peer_ack_remove: failed to remove global_image_id=647ef1ae-5a98-459b-b31b-d5aa0404c570 2023-11-24T09:02:44.653 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6800896209458998) > weight (0.25) 2023-11-24T09:02:44.654 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.35508233455961347) > weight (0.25) 2023-11-24T09:02:44.654 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5157387500053722) > weight (0.25) 2023-11-24T09:02:44.654 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3126883258760148) > weight (0.25) 2023-11-24T09:02:44.654 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-11-24T09:02:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_42 2023-11-24T09:02:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-11-24T09:02:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T09:02:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-11-24T09:02:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_42 2023-11-24T09:02:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T09:02:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_42 2023-11-24T09:02:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T09:02:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-11-24T09:02:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-11-24T09:02:47.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:47.962+0000 7f5491c7d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a5d4d2680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:47.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:02:47.962+0000 7f5498c8b640 -1 rbd::mirror::ImageReplayer: 0x558a5cbe2280 [3/d4241692-b271-471f-97e7-4e41ae0eefd4] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_43 2023-11-24T09:02:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-11-24T09:02:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T09:02:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-11-24T09:02:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_43 2023-11-24T09:02:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T09:02:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_43 2023-11-24T09:02:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T09:02:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-11-24T09:02:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-11-24T09:02:49.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:49.990+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ae50340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:49.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:49.990+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/ab341c38-aeae-40e6-8dbf-7e351d9676c7] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:50.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:50.002+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad5a340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:50.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:02:50.002+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/ab341c38-aeae-40e6-8dbf-7e351d9676c7] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_44 2023-11-24T09:02:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-11-24T09:02:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T09:02:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-11-24T09:02:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_44 2023-11-24T09:02:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T09:02:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_44 2023-11-24T09:02:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T09:02:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-11-24T09:02:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-11-24T09:02:53.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:53.018+0000 7fb5d49a0640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53002b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:53.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:02:53.018+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a52f3fb80 [3/6f744513-92ac-4212-8ca4-f89cc0e36fcf] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_45 2023-11-24T09:02:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-11-24T09:02:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T09:02:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-11-24T09:02:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_45 2023-11-24T09:02:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T09:02:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_45 2023-11-24T09:02:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T09:02:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-11-24T09:02:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-11-24T09:02:55.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:02:55.902+0000 7f298c47e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a55036c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:55.905 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:02:55.902+0000 7f2992c8b640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce3b80 [3/5b977de2-7224-446c-8040-6e733cf6a84b] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:02:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:02:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:02:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_46 2023-11-24T09:02:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-11-24T09:02:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:02:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:02:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-11-24T09:02:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:02:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_46 2023-11-24T09:02:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:02:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:02:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_46 2023-11-24T09:02:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:02:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:02:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:02:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-11-24T09:02:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-11-24T09:02:59.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:02:59.162+0000 7f298c47e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5640a5503860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:02:59.164 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:02:59.162+0000 7f2992c8b640 -1 rbd::mirror::ImageReplayer: 0x5640a4ce2f00 [3/02bfc8e5-4c3a-4991-aa98-8ab3d6782ef1] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:03:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:03:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_47 2023-11-24T09:03:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-11-24T09:03:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:03:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:03:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-11-24T09:03:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:03:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_47 2023-11-24T09:03:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:03:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_47 2023-11-24T09:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:03:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-11-24T09:03:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-11-24T09:03:02.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:03:02.178+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a538d1ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:03:02.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:03:02.178+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a52f3e280 [3/5023e532-77d0-4a81-9a30-bf38f68789b1] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:03:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:03:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_48 2023-11-24T09:03:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-11-24T09:03:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:03:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:03:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-11-24T09:03:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:03:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_48 2023-11-24T09:03:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:03:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:03:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_48 2023-11-24T09:03:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:03:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-11-24T09:03:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-11-24T09:03:05.186 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:03:05.186+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53f36ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:03:05.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:03:05.186+0000 7fb5db1ad640 -1 rbd::mirror::ImageReplayer: 0x558a5391a000 [3/934c4e48-8c50-49b0-b632-f50d1f739feb] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:03:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:03:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_49 2023-11-24T09:03:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-11-24T09:03:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:03:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:03:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-11-24T09:03:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:03:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_49 2023-11-24T09:03:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:03:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:03:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_49 2023-11-24T09:03:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:03:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-11-24T09:03:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-11-24T09:03:07.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:03:07.930+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ae50820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:03:07.931 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:03:07.930+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a000 [3/c2dd7404-8a4d-4ca3-966a-b940cfe221e9] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:03:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:03:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_50 2023-11-24T09:03:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-11-24T09:03:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:03:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local snap=snap 2023-11-24T09:03:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-11-24T09:03:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing snap: 100% complete...done. 2023-11-24T09:03:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_50 2023-11-24T09:03:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:03:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:03:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_50 2023-11-24T09:03:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:03:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-11-24T09:03:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-11-24T09:03:10.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:03:10.950+0000 7f024411a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad91a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:03:10.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:03:10.950+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959b00b180 [3/16bfa33f-6709-47f2-9afa-1242602344ab] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:03:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-24T09:03:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog TEST: image deletions should propagate 2023-11-24T09:03:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.tyy3Nh9UuT/rbd-mirror.test.log 2023-11-24T09:03:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ date +%F %T 2023-11-24T09:03:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2023-11-24 09:03:12 TEST: image deletions should propagate 2023-11-24T09:03:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2023-11-24 09:03:12 TEST: image deletions should propagate 2023-11-24T09:03:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-11-24T09:03:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_count=0 2023-11-24T09:03:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local count 2023-11-24T09:03:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local last_count=0 2023-11-24T09:03:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ true 2023-11-24T09:03:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 40 2023-11-24T09:03:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2023-11-24T09:03:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-11-24T09:03:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep images: 2023-11-24T09:03:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cut -d -f 2 2023-11-24T09:03:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T09:03:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count=0 2023-11-24T09:03:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2023-11-24T09:03:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-11-24T09:03:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state 2023-11-24T09:03:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 40 2023-11-24T09:03:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 1 -ne 1 2023-11-24T09:03:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-11-24T09:03:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep image health: 2023-11-24T09:03:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cut -d -f 3 2023-11-24T09:03:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: failed to query services: (13) Permission denied 2023-11-24T09:03:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ state=OK 2023-11-24T09:03:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = ERROR 2023-11-24T09:03:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test OK = OK 2023-11-24T09:03:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 50 2023-11-24T09:03:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2023-11-24T09:03:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-11-24T09:03:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_1 2023-11-24T09:03:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-11-24T09:03:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-11-24T09:03:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_1 deleted 2023-11-24T09:03:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_1 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-11-24T09:03:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-11-24T09:03:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_2 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:03:13.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-11-24T09:03:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-11-24T09:03:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_2 deleted 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_2 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:03:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-11-24T09:03:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_3 2023-11-24T09:03:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:03:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-11-24T09:03:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-11-24T09:03:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_3 deleted 2023-11-24T09:03:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:03:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_3 2023-11-24T09:03:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:03:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-11-24T09:03:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-11-24T09:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2023-11-24T09:03:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-11-24T09:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:03:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-24T09:03:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 deleted 2023-11-24T09:03:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:03:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:03:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:03:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:03:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-24T09:03:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_5 2023-11-24T09:03:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:03:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-11-24T09:03:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-11-24T09:03:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_5 deleted 2023-11-24T09:03:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:03:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_5 2023-11-24T09:03:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:03:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-11-24T09:03:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-11-24T09:03:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_6 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:03:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-11-24T09:03:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-11-24T09:03:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_6 deleted 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_6 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-11-24T09:03:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-11-24T09:03:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2023-11-24T09:03:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-11-24T09:03:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_7 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:03:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-11-24T09:03:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-24T09:03:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_7 deleted 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_7 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-11-24T09:03:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-24T09:03:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_8 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-11-24T09:03:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-11-24T09:03:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_8 deleted 2023-11-24T09:03:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_8 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-11-24T09:03:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-11-24T09:03:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_9 2023-11-24T09:03:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:03:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-11-24T09:03:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-11-24T09:03:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:14.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_9 deleted 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_9 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:03:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-11-24T09:03:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:03:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:03:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:03:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:03:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-11-24T09:03:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted 2023-11-24T09:03:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:03:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:03:14.851 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-11-24T09:03:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_11 2023-11-24T09:03:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-11-24T09:03:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_11 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_11 2023-11-24T09:03:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-11-24T09:03:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_11 deleted 2023-11-24T09:03:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T09:03:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_11 2023-11-24T09:03:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_11 2023-11-24T09:03:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_11 2023-11-24T09:03:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-11-24T09:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_12 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_12 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T09:03:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_12 2023-11-24T09:03:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-11-24T09:03:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_12 deleted 2023-11-24T09:03:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T09:03:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_12 2023-11-24T09:03:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_12 2023-11-24T09:03:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_12 2023-11-24T09:03:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-11-24T09:03:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_13 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_13 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T09:03:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_13 2023-11-24T09:03:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-11-24T09:03:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_13 deleted 2023-11-24T09:03:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T09:03:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_13 2023-11-24T09:03:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_13 2023-11-24T09:03:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_13 2023-11-24T09:03:15.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-11-24T09:03:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_14 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_14 2023-11-24T09:03:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T09:03:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_14 2023-11-24T09:03:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-11-24T09:03:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_14 deleted 2023-11-24T09:03:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T09:03:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_14 2023-11-24T09:03:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_14 2023-11-24T09:03:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_14 2023-11-24T09:03:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-11-24T09:03:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_15 2023-11-24T09:03:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-11-24T09:03:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_15 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_15 2023-11-24T09:03:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-11-24T09:03:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_15 deleted 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_15 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_15 2023-11-24T09:03:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_15 2023-11-24T09:03:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-11-24T09:03:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_16 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_16 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T09:03:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_16 2023-11-24T09:03:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-11-24T09:03:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_16 deleted 2023-11-24T09:03:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T09:03:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_16 2023-11-24T09:03:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_16 2023-11-24T09:03:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_16 2023-11-24T09:03:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-11-24T09:03:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_17 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_17 2023-11-24T09:03:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T09:03:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_17 2023-11-24T09:03:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.086 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-11-24T09:03:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_17 deleted 2023-11-24T09:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T09:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_17 2023-11-24T09:03:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_17 2023-11-24T09:03:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_17 2023-11-24T09:03:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-11-24T09:03:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_18 2023-11-24T09:03:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_18 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T09:03:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_18+ 2023-11-24T09:03:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-11-24T09:03:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_18 deleted 2023-11-24T09:03:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T09:03:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_18 2023-11-24T09:03:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_18 2023-11-24T09:03:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_18 2023-11-24T09:03:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_19 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T09:03:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_19 2023-11-24T09:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T09:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_19 2023-11-24T09:03:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-11-24T09:03:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_19 deleted 2023-11-24T09:03:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_19 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_19 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_19 2023-11-24T09:03:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-11-24T09:03:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_20 2023-11-24T09:03:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-11-24T09:03:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_20 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_20 2023-11-24T09:03:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-11-24T09:03:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_20 deleted 2023-11-24T09:03:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T09:03:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_20 2023-11-24T09:03:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_20 2023-11-24T09:03:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_20 2023-11-24T09:03:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-11-24T09:03:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_21 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_21 2023-11-24T09:03:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T09:03:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_21 2023-11-24T09:03:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-11-24T09:03:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_21 deleted 2023-11-24T09:03:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T09:03:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_21 2023-11-24T09:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_21 2023-11-24T09:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_21 2023-11-24T09:03:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-11-24T09:03:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_22 2023-11-24T09:03:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_22 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T09:03:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_22 2023-11-24T09:03:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-11-24T09:03:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_22 deleted 2023-11-24T09:03:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T09:03:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_22 2023-11-24T09:03:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_22 2023-11-24T09:03:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_22 2023-11-24T09:03:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-11-24T09:03:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_23 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_23 2023-11-24T09:03:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T09:03:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_23 2023-11-24T09:03:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-11-24T09:03:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_23 deleted 2023-11-24T09:03:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T09:03:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_23 2023-11-24T09:03:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_23 2023-11-24T09:03:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_23 2023-11-24T09:03:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-11-24T09:03:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_24 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_24 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T09:03:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_24 2023-11-24T09:03:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-11-24T09:03:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_24 deleted 2023-11-24T09:03:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T09:03:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_24 2023-11-24T09:03:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_24 2023-11-24T09:03:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_24 2023-11-24T09:03:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-11-24T09:03:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_25 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_25 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T09:03:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_25 2023-11-24T09:03:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-11-24T09:03:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_25 deleted 2023-11-24T09:03:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T09:03:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_25 2023-11-24T09:03:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_25 2023-11-24T09:03:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_25 2023-11-24T09:03:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-11-24T09:03:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_26 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_26 2023-11-24T09:03:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T09:03:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_26 2023-11-24T09:03:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-11-24T09:03:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_26 deleted 2023-11-24T09:03:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T09:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_26 2023-11-24T09:03:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_26 2023-11-24T09:03:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_26 2023-11-24T09:03:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-11-24T09:03:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_27 2023-11-24T09:03:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-11-24T09:03:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T09:03:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_27 2023-11-24T09:03:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T09:03:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_27 2023-11-24T09:03:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-11-24T09:03:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_27 deleted 2023-11-24T09:03:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T09:03:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_27 2023-11-24T09:03:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_27 2023-11-24T09:03:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_27 2023-11-24T09:03:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-11-24T09:03:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_28 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_28 2023-11-24T09:03:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T09:03:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_28 2023-11-24T09:03:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-11-24T09:03:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_28 deleted 2023-11-24T09:03:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T09:03:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_28 2023-11-24T09:03:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_28 2023-11-24T09:03:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_28 2023-11-24T09:03:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-11-24T09:03:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_29 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_29 2023-11-24T09:03:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T09:03:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_29 2023-11-24T09:03:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-11-24T09:03:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_29 deleted 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_29 2023-11-24T09:03:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_29 2023-11-24T09:03:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_29 2023-11-24T09:03:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-11-24T09:03:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_30 2023-11-24T09:03:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-11-24T09:03:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_30 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_30 2023-11-24T09:03:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-11-24T09:03:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_30 deleted 2023-11-24T09:03:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.692 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_30 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_30 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_30 2023-11-24T09:03:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-11-24T09:03:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_31 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_31 2023-11-24T09:03:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T09:03:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_31 2023-11-24T09:03:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-11-24T09:03:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:18.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_31 deleted 2023-11-24T09:03:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T09:03:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_31 2023-11-24T09:03:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_31 2023-11-24T09:03:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + rbd --cluster=cluster1 info mirror/image_31 2023-11-24T09:03:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-11-24T09:03:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_32 2023-11-24T09:03:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-11-24T09:03:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T09:03:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_32 2023-11-24T09:03:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T09:03:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_32 2023-11-24T09:03:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-11-24T09:03:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_32 deleted 2023-11-24T09:03:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_32 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_32 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_32 2023-11-24T09:03:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-11-24T09:03:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_33 2023-11-24T09:03:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_33 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T09:03:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_33 2023-11-24T09:03:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-11-24T09:03:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_33 deleted 2023-11-24T09:03:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_33 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_33 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_33 2023-11-24T09:03:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-11-24T09:03:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_34 2023-11-24T09:03:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-11-24T09:03:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T09:03:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_34 2023-11-24T09:03:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T09:03:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_34 2023-11-24T09:03:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-11-24T09:03:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_34 deleted 2023-11-24T09:03:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_34 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_34 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_34 2023-11-24T09:03:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-11-24T09:03:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_35 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_35 2023-11-24T09:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T09:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_35 2023-11-24T09:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-11-24T09:03:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_35 deleted 2023-11-24T09:03:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T09:03:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_35 2023-11-24T09:03:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_35 2023-11-24T09:03:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_35 2023-11-24T09:03:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-11-24T09:03:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_36 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_36 2023-11-24T09:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T09:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_36 2023-11-24T09:03:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-11-24T09:03:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_36 deleted 2023-11-24T09:03:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T09:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_36 2023-11-24T09:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_36 2023-11-24T09:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_36 2023-11-24T09:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:20.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-11-24T09:03:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_37 2023-11-24T09:03:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-11-24T09:03:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T09:03:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_37 2023-11-24T09:03:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T09:03:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_37 2023-11-24T09:03:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-11-24T09:03:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_37 deleted 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_37 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_37 2023-11-24T09:03:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_37 2023-11-24T09:03:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:20.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-11-24T09:03:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_38 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:20.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_38 2023-11-24T09:03:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T09:03:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_38 2023-11-24T09:03:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-11-24T09:03:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_38 deleted 2023-11-24T09:03:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T09:03:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_38 2023-11-24T09:03:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_38 2023-11-24T09:03:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_38 2023-11-24T09:03:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-11-24T09:03:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_39 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_39 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T09:03:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_39 2023-11-24T09:03:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-11-24T09:03:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_39 deleted 2023-11-24T09:03:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T09:03:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_39 2023-11-24T09:03:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_39 2023-11-24T09:03:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_39 2023-11-24T09:03:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-11-24T09:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_40 2023-11-24T09:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-11-24T09:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T09:03:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_40 2023-11-24T09:03:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T09:03:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_40 2023-11-24T09:03:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-11-24T09:03:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_40 deleted 2023-11-24T09:03:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T09:03:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_40 2023-11-24T09:03:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_40 2023-11-24T09:03:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_40 2023-11-24T09:03:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-11-24T09:03:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_41 2023-11-24T09:03:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-11-24T09:03:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_41 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_41 2023-11-24T09:03:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-11-24T09:03:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_41 deleted 2023-11-24T09:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T09:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_41 2023-11-24T09:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_41 2023-11-24T09:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_41 2023-11-24T09:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-11-24T09:03:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_42 2023-11-24T09:03:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-11-24T09:03:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T09:03:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_42 2023-11-24T09:03:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T09:03:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_42 2023-11-24T09:03:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-11-24T09:03:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_42 deleted 2023-11-24T09:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T09:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_42 2023-11-24T09:03:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_42 2023-11-24T09:03:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_42 2023-11-24T09:03:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-11-24T09:03:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_43 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_43 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T09:03:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_43 2023-11-24T09:03:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-11-24T09:03:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_43 deleted 2023-11-24T09:03:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T09:03:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_43 2023-11-24T09:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_43 2023-11-24T09:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_43 2023-11-24T09:03:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-11-24T09:03:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:21.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_44 2023-11-24T09:03:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-11-24T09:03:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T09:03:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_44 2023-11-24T09:03:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T09:03:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_44 2023-11-24T09:03:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-11-24T09:03:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_44 deleted 2023-11-24T09:03:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_44 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_44 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_44 2023-11-24T09:03:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-11-24T09:03:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_45 2023-11-24T09:03:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-11-24T09:03:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_45 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_45 2023-11-24T09:03:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-11-24T09:03:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_45 deleted 2023-11-24T09:03:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T09:03:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_45 2023-11-24T09:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_45 2023-11-24T09:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_45 2023-11-24T09:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-11-24T09:03:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_46 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_46 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:03:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_46 2023-11-24T09:03:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-11-24T09:03:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_46 deleted 2023-11-24T09:03:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_46 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_46 2023-11-24T09:03:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_46 2023-11-24T09:03:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-11-24T09:03:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_47 2023-11-24T09:03:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-11-24T09:03:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:03:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_47 2023-11-24T09:03:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:03:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_47 2023-11-24T09:03:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-11-24T09:03:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_47 deleted 2023-11-24T09:03:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_47 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_47 2023-11-24T09:03:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_47 2023-11-24T09:03:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-11-24T09:03:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_48 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_48 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:03:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_48 2023-11-24T09:03:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-11-24T09:03:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:22.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_48 deleted 2023-11-24T09:03:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:03:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_48 2023-11-24T09:03:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_48 2023-11-24T09:03:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_48 2023-11-24T09:03:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-11-24T09:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_49 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_49 2023-11-24T09:03:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:03:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_49 2023-11-24T09:03:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-11-24T09:03:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_49 deleted 2023-11-24T09:03:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:03:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_49 2023-11-24T09:03:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_49 2023-11-24T09:03:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_49 2023-11-24T09:03:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:23.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-11-24T09:03:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_50 2023-11-24T09:03:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-11-24T09:03:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:03:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:03:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_50 2023-11-24T09:03:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:03:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_50 2023-11-24T09:03:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-11-24T09:03:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:03:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_50 deleted 2023-11-24T09:03:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:03:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:03:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_50 2023-11-24T09:03:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_50 2023-11-24T09:03:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_50 2023-11-24T09:03:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-11-24T09:03:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:03:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:03:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog TEST: delete images during bootstrap 2023-11-24T09:03:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ tee -a /tmp/tmp.tyy3Nh9UuT/rbd-mirror.test.log 2023-11-24T09:03:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ date +%F %T 2023-11-24T09:03:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2023-11-24 09:03:23 TEST: delete images during bootstrap 2023-11-24T09:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2023-11-24 09:03:23 TEST: delete images during bootstrap 2023-11-24T09:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-11-24T09:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=image 2023-11-24T09:03:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-11-24T09:03:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd: mirroring is already configured for image mode 2023-11-24T09:03:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-11-24T09:03:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=image 2023-11-24T09:03:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-11-24T09:03:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd: mirroring is already configured for image mode 2023-11-24T09:03:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ start_mirror cluster1 2023-11-24T09:03:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance 2023-11-24T09:03:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set_cluster_instance cluster1 cluster instance 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local val=cluster1 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster_var_name=cluster 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local instance_var_name=instance 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=cluster1 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ cluster1 = cluster1 ] 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval cluster=cluster1 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cluster=cluster1 2023-11-24T09:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ eval instance=0 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ instance=0 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=test 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 1 10 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_1 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_1 512M 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:03:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:03:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:03:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:03:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:03:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:03:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2023-11-24T09:03:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_1 2023-11-24T09:03:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:03:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-11-24T09:03:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:03:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-11-24T09:03:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_1': 2023-11-24T09:03:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:03:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:03:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:03:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 18ad59c30f0a 2023-11-24T09:03:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.18ad59c30f0a 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:03:23 2023 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:03:23 2023 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:03:23 2023 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: aecbf762-6272-4e8d-ade4-f7504f36c7c4 2023-11-24T09:03:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:03:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_1 2023-11-24T09:03:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:03:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T09:03:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:03:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:03:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 62485504 bytes 2023-11-24T09:03:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 113437696 bytes 2023-11-24T09:03:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 156080128 bytes 2023-11-24T09:03:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 166687744 bytes 2023-11-24T09:03:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 212460544 bytes 2023-11-24T09:03:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 228755456 bytes 2023-11-24T09:03:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 270894080 bytes 2023-11-24T09:03:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 306892800 bytes 2023-11-24T09:03:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 315721728 bytes 2023-11-24T09:03:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 397128704 bytes 2023-11-24T09:03:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 420049920 bytes 2023-11-24T09:03:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 420665344 bytes 2023-11-24T09:03:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 470990848 bytes 2023-11-24T09:03:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 479299584 bytes 2023-11-24T09:03:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 489329664 bytes 2023-11-24T09:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 530463744 bytes 2023-11-24T09:03:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 541204480 bytes 2023-11-24T09:03:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 555336704 bytes 2023-11-24T09:03:27.433 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 567089152 bytes 2023-11-24T09:03:27.562 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 578621440 bytes 2023-11-24T09:03:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 638328832 bytes 2023-11-24T09:03:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 702314496 bytes 2023-11-24T09:03:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 723222528 bytes 2023-11-24T09:03:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 823145472 bytes 2023-11-24T09:03:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 832051200 bytes 2023-11-24T09:03:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 909777920 bytes 2023-11-24T09:03:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 964305920 bytes 2023-11-24T09:03:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1014088704 bytes 2023-11-24T09:03:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1035635712 bytes 2023-11-24T09:03:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1128967168 bytes 2023-11-24T09:03:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1182532608 bytes 2023-11-24T09:03:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1204703232 bytes 2023-11-24T09:03:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1263350784 bytes 2023-11-24T09:03:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1343021056 bytes 2023-11-24T09:03:30.895 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1416948736 bytes 2023-11-24T09:03:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:03:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_1 2023-11-24T09:03:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-11-24T09:03:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=12b0438af692 2023-11-24T09:03:30.953 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:03:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_1 present 12b0438af692 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=12b0438af692 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_1 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-11-24T09:03:31.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:03:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=12b0438af692 2023-11-24T09:03:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 12b0438af692 2023-11-24T09:03:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 12b0438af692 -o 12b0438af692 = 12b0438af692 2023-11-24T09:03:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:03:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:03:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:03:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_1 2023-11-24T09:03:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:03:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:03:31.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:03:31.114 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-11-24T09:03:50.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:03:50.785+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53f36820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:03:50.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:03:50.785+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a5391a000 [3/aecbf762-6272-4e8d-ade4-f7504f36c7c4] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:03:50.842 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9991827671185693) > weight (0.25) 2023-11-24T09:03:50.842 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-24T09:03:50.842 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-24T09:03:50.842 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-11-24T09:03:50.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:03:50.841+0000 7f299348c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 66056) UID: 1000 2023-11-24T09:03:55.843 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-24T09:03:55.843 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:03:55.844 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-24T09:03:55.844 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-24T09:03:55.844 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-24T09:03:55.847 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-24T09:03:55.847 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-11-24T09:04:00.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:04:00.833+0000 7f349b6fa640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fd969a5c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-11-24T09:04:00.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:04:00.833+0000 7f349b6fa640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fd969a5c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d8222b82-a76f-4f5f-904a-6a59fe53ff7b, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-11-24T09:04:00.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:04:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-11-24T09:04:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:04:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:04:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:04:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:04:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:04:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_1 2023-11-24T09:04:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:04:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:04:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: --cluster=cluster1 info mirror/image_1 2023-11-24T09:04:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=12b0438af692 2023-11-24T09:04:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_1 deleted 12b0438af692 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=12b0438af692 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_1 2023-11-24T09:04:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:04:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-11-24T09:04:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:04:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-11-24T09:04:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:04:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:04:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:04:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:04:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_1 2023-11-24T09:04:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:04:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-11-24T09:04:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_1 2023-11-24T09:04:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:04:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:04:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_1 2023-11-24T09:04:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_1 2023-11-24T09:04:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-11-24T09:04:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-11-24T09:04:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_2 2023-11-24T09:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_2 512M 2023-11-24T09:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:04:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:04:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:04:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:04:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:04:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2023-11-24T09:04:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_2 2023-11-24T09:04:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:04:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-11-24T09:04:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:04:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-11-24T09:04:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_2': 2023-11-24T09:04:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:04:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:04:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:04:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 18c76368ce15 2023-11-24T09:04:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.18c76368ce15 2023-11-24T09:04:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:04:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:04:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:04:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:04:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:04:01 2023 2023-11-24T09:04:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:04:01 2023 2023-11-24T09:04:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:04:01 2023 2023-11-24T09:04:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:04:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:04:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 0a49d51e-0146-4a4f-90e4-75d143e44d14 2023-11-24T09:04:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:04:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_2 2023-11-24T09:04:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:04:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2023-11-24T09:04:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:04:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:04:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 25262080 bytes 2023-11-24T09:04:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 145426432 bytes 2023-11-24T09:04:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 229520384 bytes 2023-11-24T09:04:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 269946880 bytes 2023-11-24T09:04:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 323542016 bytes 2023-11-24T09:04:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 341745664 bytes 2023-11-24T09:04:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 378348544 bytes 2023-11-24T09:04:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 415499264 bytes 2023-11-24T09:04:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 426452992 bytes 2023-11-24T09:04:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 477519872 bytes 2023-11-24T09:04:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 506583040 bytes 2023-11-24T09:04:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 527547392 bytes 2023-11-24T09:04:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 557033472 bytes 2023-11-24T09:04:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 608661504 bytes 2023-11-24T09:04:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 676016128 bytes 2023-11-24T09:04:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 728077312 bytes 2023-11-24T09:04:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 811432960 bytes 2023-11-24T09:04:06.806 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 897087488 bytes 2023-11-24T09:04:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 944252928 bytes 2023-11-24T09:04:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 979064832 bytes 2023-11-24T09:04:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1030759424 bytes 2023-11-24T09:04:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1077548032 bytes 2023-11-24T09:04:07.655 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-24T09:04:07.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-24T09:04:07.655 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-24T09:04:07.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:04:07.653+0000 7f32ab71c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 64950) UID: 1000 2023-11-24T09:04:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1169976320 bytes 2023-11-24T09:04:08.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1263420416 bytes 2023-11-24T09:04:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1308662784 bytes 2023-11-24T09:04:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1318424576 bytes 2023-11-24T09:04:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1406595072 bytes 2023-11-24T09:04:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1479972864 bytes 2023-11-24T09:04:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1524421632 bytes 2023-11-24T09:04:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1594198016 bytes 2023-11-24T09:04:09.656 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-24T09:04:09.656 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:04:09.657 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-24T09:04:09.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-24T09:04:09.657 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-24T09:04:09.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-24T09:04:09.659 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-11-24T09:04:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1633925120 bytes 2023-11-24T09:04:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1689877504 bytes 2023-11-24T09:04:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1715815424 bytes 2023-11-24T09:04:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1753096192 bytes 2023-11-24T09:04:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1817886720 bytes 2023-11-24T09:04:10.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1847859200 bytes 2023-11-24T09:04:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1918422016 bytes 2023-11-24T09:04:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1947724800 bytes 2023-11-24T09:04:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1983484928 bytes 2023-11-24T09:04:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 2024171520 bytes 2023-11-24T09:04:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 2113117184 bytes 2023-11-24T09:04:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2188684288 bytes 2023-11-24T09:04:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2230035456 bytes 2023-11-24T09:04:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2230651904 bytes 2023-11-24T09:04:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2231283712 bytes 2023-11-24T09:04:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2249095168 bytes 2023-11-24T09:04:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2276147200 bytes 2023-11-24T09:04:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2321312768 bytes 2023-11-24T09:04:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2376457216 bytes 2023-11-24T09:04:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2434307072 bytes 2023-11-24T09:04:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2476583936 bytes 2023-11-24T09:04:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:04:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:04:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:04:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:04:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_2 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-11-24T09:04:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:04:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=12b046a76b62 2023-11-24T09:04:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:04:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_2 present 12b046a76b62 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=12b046a76b62 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_2 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-11-24T09:04:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:04:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=12b046a76b62 2023-11-24T09:04:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 12b046a76b62 2023-11-24T09:04:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 12b046a76b62 -o 12b046a76b62 = 12b046a76b62 2023-11-24T09:04:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:04:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:04:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:04:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_2 2023-11-24T09:04:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-11-24T09:04:20.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:04:20.785+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a53f36d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:04:20.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:04:20.785+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a5391a000 [3/0a49d51e-0146-4a4f-90e4-75d143e44d14] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:04:44.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:04:44.832+0000 7f349befb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fd96a7dc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-11-24T09:04:44.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:04:44.832+0000 7f349befb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fd96a7dc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d8222b82-a76f-4f5f-904a-6a59fe53ff7b, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-11-24T09:04:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:04:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-11-24T09:04:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:04:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:04:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:04:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:04:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_2 2023-11-24T09:04:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-11-24T09:04:44.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:04:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-11-24T09:04:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:04:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_2 deleted 2023-11-24T09:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_2 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-11-24T09:04:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:04:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-11-24T09:04:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:04:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:04:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:04:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:04:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_2 2023-11-24T09:04:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-11-24T09:04:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_2 2023-11-24T09:04:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:04:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_2 2023-11-24T09:04:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_2 2023-11-24T09:04:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-11-24T09:04:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-11-24T09:04:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_3 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_3 512M 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:04:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:04:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:04:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:04:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2023-11-24T09:04:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_3 2023-11-24T09:04:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:04:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:04:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-11-24T09:04:46.668 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:04:46.664+0000 7f42b0626640 -1 reset not still connected to 0x561a8934a820 2023-11-24T09:04:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:04:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-11-24T09:04:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_3': 2023-11-24T09:04:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:04:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:04:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:04:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 18de3ab20ec2 2023-11-24T09:04:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.18de3ab20ec2 2023-11-24T09:04:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:04:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:04:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:04:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:04:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:04:45 2023 2023-11-24T09:04:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:04:45 2023 2023-11-24T09:04:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:04:45 2023 2023-11-24T09:04:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:04:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:04:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 0e67e83b-cc76-4cbd-84dc-15ebaff638ff 2023-11-24T09:04:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:04:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_3 2023-11-24T09:04:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:04:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:04:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T09:04:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:04:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:04:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 583680 bytes 2023-11-24T09:04:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 27592704 bytes 2023-11-24T09:04:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 67296256 bytes 2023-11-24T09:04:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 83831808 bytes 2023-11-24T09:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 101353472 bytes 2023-11-24T09:04:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 111735808 bytes 2023-11-24T09:04:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 129908736 bytes 2023-11-24T09:04:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 181492736 bytes 2023-11-24T09:04:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 226802688 bytes 2023-11-24T09:04:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 268141568 bytes 2023-11-24T09:04:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 307527680 bytes 2023-11-24T09:04:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 334474240 bytes 2023-11-24T09:04:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 335099904 bytes 2023-11-24T09:04:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 382532608 bytes 2023-11-24T09:04:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 491545600 bytes 2023-11-24T09:04:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 550469632 bytes 2023-11-24T09:04:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 636652544 bytes 2023-11-24T09:04:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 660908032 bytes 2023-11-24T09:04:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 698293248 bytes 2023-11-24T09:04:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 743800832 bytes 2023-11-24T09:04:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 785013760 bytes 2023-11-24T09:04:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 832872448 bytes 2023-11-24T09:04:51.196 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 880617472 bytes 2023-11-24T09:04:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 901441536 bytes 2023-11-24T09:04:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 927915008 bytes 2023-11-24T09:04:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 950233088 bytes 2023-11-24T09:04:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 995275776 bytes 2023-11-24T09:04:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1025764352 bytes 2023-11-24T09:04:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1062449152 bytes 2023-11-24T09:04:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1102255104 bytes 2023-11-24T09:04:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1157130240 bytes 2023-11-24T09:04:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1214830592 bytes 2023-11-24T09:04:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1238922240 bytes 2023-11-24T09:04:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1276949504 bytes 2023-11-24T09:04:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1289859072 bytes 2023-11-24T09:04:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1336891392 bytes 2023-11-24T09:04:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1398631424 bytes 2023-11-24T09:04:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:04:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:04:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:04:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:04:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-11-24T09:04:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:04:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:04:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:04:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:04:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:04:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_3 2023-11-24T09:04:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:04:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-11-24T09:04:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:04:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=12b03c43f9b1 2023-11-24T09:04:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_3 present 12b03c43f9b1 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=12b03c43f9b1 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_3 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:04:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:04:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-11-24T09:04:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:04:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=12b03c43f9b1 2023-11-24T09:04:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 12b03c43f9b1 2023-11-24T09:04:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 12b03c43f9b1 -o 12b03c43f9b1 = 12b03c43f9b1 2023-11-24T09:04:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:04:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:04:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:04:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_3 2023-11-24T09:04:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:04:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:04:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:04:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-11-24T09:05:20.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:05:20.784+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a537956c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:05:20.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:05:20.784+0000 7fb5db1ad640 -1 rbd::mirror::ImageReplayer: 0x558a5391a000 [3/0e67e83b-cc76-4cbd-84dc-15ebaff638ff] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:05:23.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:05:23.708+0000 7f349befb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fd96601000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-11-24T09:05:23.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:05:23.708+0000 7f349befb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fd96601000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d8222b82-a76f-4f5f-904a-6a59fe53ff7b, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-11-24T09:05:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:05:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-11-24T09:05:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:05:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:05:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:05:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:05:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:05:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:05:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_3 2023-11-24T09:05:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:05:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:05:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + rbd --cluster=cluster1 info mirror/image_3sed 2023-11-24T09:05:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:05:23.848 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-11-24T09:05:23.849 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-11-24T09:05:23.849 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-24T09:05:23.850 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:05:23.848+0000 7f549948c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 68894) UID: 1000 2023-11-24T09:05:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-11-24T09:05:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:05:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:05:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_3 deleted 2023-11-24T09:05:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:05:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:05:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:05:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:05:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:05:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:05:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_3 2023-11-24T09:05:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:05:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:05:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-11-24T09:05:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:05:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-11-24T09:05:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:05:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:05:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:05:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:05:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_3 2023-11-24T09:05:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:05:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:05:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-11-24T09:05:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_3 2023-11-24T09:05:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:05:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:05:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:05:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_3 2023-11-24T09:05:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:05:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_3 2023-11-24T09:05:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-11-24T09:05:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-11-24T09:05:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:05:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:05:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_4 2023-11-24T09:05:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_4 512M 2023-11-24T09:05:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:05:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:05:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:05:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:05:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:05:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:05:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:05:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:05:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:05:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2023-11-24T09:05:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_4 2023-11-24T09:05:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:05:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:05:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:05:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-11-24T09:05:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:05:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-11-24T09:05:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_4': 2023-11-24T09:05:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:05:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:05:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:05:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 18f48b98b6d 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.18f48b98b6d 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:05:24 2023 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:05:24 2023 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:05:24 2023 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:05:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 2583bd7f-f6fe-4fb1-9928-7e70734640bf 2023-11-24T09:05:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:05:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_4 2023-11-24T09:05:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:05:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:05:24.755 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:05:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=9 2023-11-24T09:05:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:05:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:05:24.878 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T09:05:24.870+0000 7f4b5a023640 -1 reset not still connected to 0x56021eacc410 2023-11-24T09:05:24.879 INFO:tasks.ceph.cluster1.osd.2.smithi153.stderr:2023-11-24T09:05:24.870+0000 7f4b5a023640 -1 reset not still connected to 0x56021eacc8f0 2023-11-24T09:05:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 18996224 bytes 2023-11-24T09:05:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 70811648 bytes 2023-11-24T09:05:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 109841408 bytes 2023-11-24T09:05:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 148687872 bytes 2023-11-24T09:05:25.849 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-11-24T09:05:25.850 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:05:25.850 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-11-24T09:05:25.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-11-24T09:05:25.851 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-11-24T09:05:25.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-11-24T09:05:25.854 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2023-11-24T09:05:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 264216576 bytes 2023-11-24T09:05:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 298088448 bytes 2023-11-24T09:05:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 354185216 bytes 2023-11-24T09:05:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 420463616 bytes 2023-11-24T09:05:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 421091328 bytes 2023-11-24T09:05:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 478817280 bytes 2023-11-24T09:05:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 502768640 bytes 2023-11-24T09:05:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 522312704 bytes 2023-11-24T09:05:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 599371776 bytes 2023-11-24T09:05:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 626733056 bytes 2023-11-24T09:05:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 672902144 bytes 2023-11-24T09:05:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 722748416 bytes 2023-11-24T09:05:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 793458688 bytes 2023-11-24T09:05:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 808208384 bytes 2023-11-24T09:05:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 842915840 bytes 2023-11-24T09:05:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 950302720 bytes 2023-11-24T09:05:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1035153408 bytes 2023-11-24T09:05:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1072392192 bytes 2023-11-24T09:05:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1083621376 bytes 2023-11-24T09:05:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1137774592 bytes 2023-11-24T09:05:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1203623936 bytes 2023-11-24T09:05:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1208690688 bytes 2023-11-24T09:05:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1253882880 bytes 2023-11-24T09:05:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1292127232 bytes 2023-11-24T09:05:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1313718272 bytes 2023-11-24T09:05:31.203 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1350682624 bytes 2023-11-24T09:05:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1399158784 bytes 2023-11-24T09:05:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1466557440 bytes 2023-11-24T09:05:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1517150208 bytes 2023-11-24T09:05:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1578372096 bytes 2023-11-24T09:05:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1643064320 bytes 2023-11-24T09:05:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1658848256 bytes 2023-11-24T09:05:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1668340736 bytes 2023-11-24T09:05:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1731197952 bytes 2023-11-24T09:05:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1761192960 bytes 2023-11-24T09:05:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1828175872 bytes 2023-11-24T09:05:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1851146240 bytes 2023-11-24T09:05:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:05:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:05:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:05:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:05:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-11-24T09:05:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:05:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:05:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:05:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:05:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:05:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:05:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:05:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:05:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:05:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:05:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:05:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=156eb5200131 2023-11-24T09:05:33.819 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:05:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 present 156eb5200131 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=156eb5200131 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:05:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:05:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:05:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=156eb5200131 2023-11-24T09:05:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 156eb5200131 2023-11-24T09:05:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 156eb5200131 -o 156eb5200131 = 156eb5200131 2023-11-24T09:05:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:05:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:05:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:05:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_4 2023-11-24T09:05:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:05:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:05:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:05:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-11-24T09:05:49.660 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6628273208453225) > weight (0.25) 2023-11-24T09:05:49.660 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-24T09:05:49.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-24T09:05:49.661 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-24T09:05:49.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:05:49.659+0000 7f331ea5f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 63000) UID: 1000 2023-11-24T09:05:49.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:05:49.991+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959aa99ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:05:49.995 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:05:49.991+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/2583bd7f-f6fe-4fb1-9928-7e70734640bf] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:05:50.661 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-24T09:05:50.661 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:05:50.662 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-24T09:05:50.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-24T09:05:50.662 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-24T09:05:50.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-24T09:05:50.666 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2023-11-24T09:06:03.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:06:03.919+0000 7f34a8f15640 -1 rbd::mirror::ImageMap: 0x55fd95e92b40 handle_peer_ack_remove: failed to remove global_image_id=2583bd7f-f6fe-4fb1-9928-7e70734640bf 2023-11-24T09:06:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-11-24T09:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:06:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:06:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:06:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:06:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:06:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=156eb5200131 2023-11-24T09:06:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:06:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 deleted 156eb5200131 2023-11-24T09:06:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=156eb5200131 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:06:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:06:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=156eb5200131 2023-11-24T09:06:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 156eb5200131 2023-11-24T09:06:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 156eb5200131 -o 156eb5200131 = 156eb5200131 2023-11-24T09:06:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:06:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:06:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:06:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 deleted 156eb5200131 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=156eb5200131 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:06:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:06:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=156eb5200131 2023-11-24T09:06:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 156eb5200131 2023-11-24T09:06:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 156eb5200131 -o 156eb5200131 = 156eb5200131 2023-11-24T09:06:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:06:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:06:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T09:06:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 deleted 156eb5200131 2023-11-24T09:06:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:06:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=156eb5200131 2023-11-24T09:06:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:06:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:06:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:06:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:06:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:06:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=156eb5200131 2023-11-24T09:06:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 156eb5200131 2023-11-24T09:06:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 156eb5200131 -o 156eb5200131 = 156eb5200131 2023-11-24T09:06:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:06:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:06:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T09:06:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 deleted 156eb5200131 2023-11-24T09:06:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:06:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=156eb5200131 2023-11-24T09:06:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:06:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:06:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:06:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:06:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:06:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=156eb5200131 2023-11-24T09:06:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 156eb5200131 2023-11-24T09:06:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 156eb5200131 -o 156eb5200131 = 156eb5200131 2023-11-24T09:06:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:06:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:06:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T09:06:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 deleted 156eb5200131 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=156eb5200131 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:06:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:06:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=156eb5200131 2023-11-24T09:06:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 156eb5200131 2023-11-24T09:06:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 156eb5200131 -o 156eb5200131 = 156eb5200131 2023-11-24T09:06:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:06:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:06:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T09:06:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_4 deleted 156eb5200131 2023-11-24T09:06:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:06:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=156eb5200131 2023-11-24T09:06:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:06:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:06:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_4 2023-11-24T09:06:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-11-24T09:06:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:06:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-24T09:06:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:06:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:06:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:06:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:06:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_4 2023-11-24T09:06:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:06:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-11-24T09:06:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_4 2023-11-24T09:06:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:06:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:06:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_4 2023-11-24T09:06:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:06:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_4 2023-11-24T09:06:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-11-24T09:06:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-11-24T09:06:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:06:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:06:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_5 2023-11-24T09:06:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_5 512M 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:06:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2023-11-24T09:06:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_5 2023-11-24T09:06:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:06:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:06:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:06:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-11-24T09:06:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:06:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-11-24T09:06:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_5': 2023-11-24T09:06:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:06:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 19189999d75 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.19189999d75 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:06:28 2023 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:06:28 2023 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:06:28 2023 2023-11-24T09:06:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:06:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:06:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 4cd9c679-deb0-4d6c-89b6-26a7bdb25d94 2023-11-24T09:06:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:06:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_5 2023-11-24T09:06:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:06:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:06:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:06:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2023-11-24T09:06:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:06:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:06:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 76555264 bytes 2023-11-24T09:06:29.564 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 77180928 bytes 2023-11-24T09:06:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 129596416 bytes 2023-11-24T09:06:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 186940416 bytes 2023-11-24T09:06:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 215917568 bytes 2023-11-24T09:06:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 233658368 bytes 2023-11-24T09:06:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 242985984 bytes 2023-11-24T09:06:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 261865472 bytes 2023-11-24T09:06:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 300284928 bytes 2023-11-24T09:06:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 344707072 bytes 2023-11-24T09:06:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 372841472 bytes 2023-11-24T09:06:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 415754240 bytes 2023-11-24T09:06:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 486462464 bytes 2023-11-24T09:06:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 515202048 bytes 2023-11-24T09:06:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 544492544 bytes 2023-11-24T09:06:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 604160000 bytes 2023-11-24T09:06:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 653091840 bytes 2023-11-24T09:06:32.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 714645504 bytes 2023-11-24T09:06:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 773781504 bytes 2023-11-24T09:06:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 834288640 bytes 2023-11-24T09:06:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 892441600 bytes 2023-11-24T09:06:33.479 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 945568768 bytes 2023-11-24T09:06:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 960310272 bytes 2023-11-24T09:06:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1058414592 bytes 2023-11-24T09:06:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1077395456 bytes 2023-11-24T09:06:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1109896192 bytes 2023-11-24T09:06:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1126546432 bytes 2023-11-24T09:06:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1127189504 bytes 2023-11-24T09:06:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1147444224 bytes 2023-11-24T09:06:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1183295488 bytes 2023-11-24T09:06:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1194836992 bytes 2023-11-24T09:06:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:06:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:06:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_5 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:06:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-11-24T09:06:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:06:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=12b02f8a22f1 2023-11-24T09:06:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_5 present 12b02f8a22f1 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=12b02f8a22f1 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_5 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:06:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:06:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-11-24T09:06:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:06:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=12b02f8a22f1 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 12b02f8a22f1 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 12b02f8a22f1 -o 12b02f8a22f1 = 12b02f8a22f1 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_5 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:06:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-11-24T09:06:50.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:06:50.786+0000 7fb5d49a0640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a537944e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:06:50.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:06:50.786+0000 7fb5da9ac640 -1 rbd::mirror::ImageReplayer: 0x558a52f3f400 [3/4cd9c679-deb0-4d6c-89b6-26a7bdb25d94] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:07:02.854 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-11-24T09:07:02.855 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-11-24T09:07:02.855 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-11-24T09:07:02.855 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-11-24T09:07:02.855 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:07:02.856 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:07:02.854+0000 7fd7c6c78640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 74370) UID: 1000 2023-11-24T09:07:06.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:07:06.894+0000 7f349befb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fd96601c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-11-24T09:07:06.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:07:06.894+0000 7f349befb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fd96601c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d8222b82-a76f-4f5f-904a-6a59fe53ff7b, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-11-24T09:07:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:07:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-11-24T09:07:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:07:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:07:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:07:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:07:06.962 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:07:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_5 2023-11-24T09:07:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:07:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:07:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-11-24T09:07:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-11-24T09:07:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:07:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:07:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_5 deleted 2023-11-24T09:07:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_5 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:07:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-11-24T09:07:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:07:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-11-24T09:07:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:07:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:07:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:07:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:07:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_5 2023-11-24T09:07:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:07:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-11-24T09:07:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_5 2023-11-24T09:07:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:07:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:07:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_5 2023-11-24T09:07:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_5 2023-11-24T09:07:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-11-24T09:07:07.343 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-11-24T09:07:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:07:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:07:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_6 2023-11-24T09:07:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_6 512M 2023-11-24T09:07:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:07:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:07:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:07:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:07:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:07:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:07:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:07:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2023-11-24T09:07:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_6 2023-11-24T09:07:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:07:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-11-24T09:07:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:07:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-11-24T09:07:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_6': 2023-11-24T09:07:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:07:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:07:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:07:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 192dae33906a 2023-11-24T09:07:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.192dae33906a 2023-11-24T09:07:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:07:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:07:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:07:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:07:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:07:07 2023 2023-11-24T09:07:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:07:07 2023 2023-11-24T09:07:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:07:07 2023 2023-11-24T09:07:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:07:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:07:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 58a4531f-fa84-4be1-bbec-019b835dfab7 2023-11-24T09:07:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:07:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_6 2023-11-24T09:07:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:07:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=14 2023-11-24T09:07:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:07:08.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:07:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 63355904 bytes 2023-11-24T09:07:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 140349440 bytes 2023-11-24T09:07:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 169585664 bytes 2023-11-24T09:07:08.957 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-11-24T09:07:08.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-11-24T09:07:08.958 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-11-24T09:07:08.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-11-24T09:07:08.960 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-11-24T09:07:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 194862080 bytes 2023-11-24T09:07:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 213053440 bytes 2023-11-24T09:07:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 247805952 bytes 2023-11-24T09:07:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 278572032 bytes 2023-11-24T09:07:09.757 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 313654272 bytes 2023-11-24T09:07:09.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 327425024 bytes 2023-11-24T09:07:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 397646848 bytes 2023-11-24T09:07:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 455193600 bytes 2023-11-24T09:07:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 477404160 bytes 2023-11-24T09:07:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 506744832 bytes 2023-11-24T09:07:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 524194816 bytes 2023-11-24T09:07:10.960 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 531549184 bytes 2023-11-24T09:07:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 587894784 bytes 2023-11-24T09:07:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 643911680 bytes 2023-11-24T09:07:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 694810624 bytes 2023-11-24T09:07:11.779 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 726133760 bytes 2023-11-24T09:07:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 760433664 bytes 2023-11-24T09:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 813036544 bytes 2023-11-24T09:07:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 850504704 bytes 2023-11-24T09:07:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 869080064 bytes 2023-11-24T09:07:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 928433152 bytes 2023-11-24T09:07:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 962089984 bytes 2023-11-24T09:07:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1018403840 bytes 2023-11-24T09:07:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1065374720 bytes 2023-11-24T09:07:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1081431040 bytes 2023-11-24T09:07:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1150519296 bytes 2023-11-24T09:07:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1231095808 bytes 2023-11-24T09:07:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1272612864 bytes 2023-11-24T09:07:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1335908352 bytes 2023-11-24T09:07:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1406796800 bytes 2023-11-24T09:07:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1415380992 bytes 2023-11-24T09:07:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1416518656 bytes 2023-11-24T09:07:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1440960512 bytes 2023-11-24T09:07:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1473112064 bytes 2023-11-24T09:07:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1527631872 bytes 2023-11-24T09:07:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1613885440 bytes 2023-11-24T09:07:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1675755520 bytes 2023-11-24T09:07:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1715934208 bytes 2023-11-24T09:07:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1788562432 bytes 2023-11-24T09:07:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1826198528 bytes 2023-11-24T09:07:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1910333440 bytes 2023-11-24T09:07:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1940225024 bytes 2023-11-24T09:07:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1955939328 bytes 2023-11-24T09:07:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1980950528 bytes 2023-11-24T09:07:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2046356480 bytes 2023-11-24T09:07:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2066040832 bytes 2023-11-24T09:07:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2139470848 bytes 2023-11-24T09:07:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2162947072 bytes 2023-11-24T09:07:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2185593856 bytes 2023-11-24T09:07:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2244963328 bytes 2023-11-24T09:07:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2290101248 bytes 2023-11-24T09:07:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2311810048 bytes 2023-11-24T09:07:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2336117760 bytes 2023-11-24T09:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2362971136 bytes 2023-11-24T09:07:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2429251584 bytes 2023-11-24T09:07:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2484389888 bytes 2023-11-24T09:07:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2553203712 bytes 2023-11-24T09:07:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2581316608 bytes 2023-11-24T09:07:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2593024000 bytes 2023-11-24T09:07:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2597554176 bytes 2023-11-24T09:07:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2728390656 bytes 2023-11-24T09:07:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2744837120 bytes 2023-11-24T09:07:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2773330944 bytes 2023-11-24T09:07:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2818298880 bytes 2023-11-24T09:07:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2860114944 bytes 2023-11-24T09:07:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2887255040 bytes 2023-11-24T09:07:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2914485248 bytes 2023-11-24T09:07:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:07:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:07:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:07:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:07:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-11-24T09:07:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:07:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:07:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:07:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:07:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_6 2023-11-24T09:07:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-11-24T09:07:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:07:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=168460710310 2023-11-24T09:07:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:07:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_6 present 168460710310 2023-11-24T09:07:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:07:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=168460710310 2023-11-24T09:07:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:07:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:07:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_6 2023-11-24T09:07:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-11-24T09:07:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:07:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=168460710310 2023-11-24T09:07:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 168460710310 2023-11-24T09:07:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 168460710310 -o 168460710310 = 168460710310 2023-11-24T09:07:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:07:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:07:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:07:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_6 2023-11-24T09:07:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-11-24T09:07:28.667 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8741826275913059) > weight (0.25) 2023-11-24T09:07:28.668 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7603656036090597) > weight (0.25) 2023-11-24T09:07:28.668 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5546019807627478) > weight (0.25) 2023-11-24T09:07:28.668 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-11-24T09:07:28.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-11-24T09:07:28.668 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-11-24T09:07:28.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:07:28.666+0000 7f34a9716640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 52601) UID: 1000 2023-11-24T09:07:29.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:07:29.870+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ae51380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:07:29.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:07:29.870+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/58a4531f-fa84-4be1-bbec-019b835dfab7] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:07:32.669 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-11-24T09:07:32.669 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:07:32.670 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-11-24T09:07:32.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-11-24T09:07:32.670 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-11-24T09:07:32.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-11-24T09:07:32.673 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-11-24T09:07:49.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:07:49.993+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ae50820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:07:49.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:07:49.993+0000 7f024a126640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/58a4531f-fa84-4be1-bbec-019b835dfab7] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:07:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:07:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-11-24T09:07:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:07:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:07:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:07:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:07:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_6 2023-11-24T09:07:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + rbd --cluster=cluster1 info mirror/image_6 2023-11-24T09:07:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:07:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-11-24T09:07:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:07:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:07:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_6 deleted 2023-11-24T09:07:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:07:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:07:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:07:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:07:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_6 2023-11-24T09:07:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-11-24T09:07:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:07:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-11-24T09:07:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:07:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:07:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:07:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:07:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_6 2023-11-24T09:07:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-11-24T09:07:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_6 2023-11-24T09:07:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:07:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_6 2023-11-24T09:07:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_6 2023-11-24T09:07:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-11-24T09:07:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-11-24T09:07:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:07:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:07:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_7 2023-11-24T09:07:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_7 512M 2023-11-24T09:07:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:07:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:07:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:07:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:07:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:07:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:07:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:07:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:07:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2023-11-24T09:07:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_7 2023-11-24T09:07:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:07:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:07:51.712 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-11-24T09:07:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:07:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-11-24T09:07:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_7': 2023-11-24T09:07:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:07:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:07:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 194f54c40e81 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.194f54c40e81 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:07:51 2023 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:07:51 2023 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:07:51 2023 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 1ef70dba-0221-48db-a7a4-aef3a0f822d5 2023-11-24T09:07:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:07:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_7 2023-11-24T09:07:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:07:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:07:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:07:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=6 2023-11-24T09:07:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:07:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:07:53.050 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T09:07:53.049+0000 7fceae979640 -1 bdev(0x55ea72822e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-11-24T09:07:53.053 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fceae979640 time 2023-11-24T09:07:53.051612+0000 2023-11-24T09:07:53.053 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-11-24T09:07:53.053 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev) 2023-11-24T09:07:53.053 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55ea6f1344d0] 2023-11-24T09:07:53.053 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x55ea6fa10fac] 2023-11-24T09:07:53.053 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 3: ceph-osd(+0x15b3861) [0x55ea6fa11861] 2023-11-24T09:07:53.053 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcebc7c2b43] 2023-11-24T09:07:53.054 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcebc854a00] 2023-11-24T09:07:53.054 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T09:07:53.049+0000 7fceae979640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fceae979640 time 2023-11-24T09:07:53.051612+0000 2023-11-24T09:07:53.054 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-11-24T09:07:53.054 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.054 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev) 2023-11-24T09:07:53.054 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55ea6f1344d0] 2023-11-24T09:07:53.054 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x55ea6fa10fac] 2023-11-24T09:07:53.054 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 3: ceph-osd(+0x15b3861) [0x55ea6fa11861] 2023-11-24T09:07:53.054 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcebc7c2b43] 2023-11-24T09:07:53.055 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcebc854a00] 2023-11-24T09:07:53.055 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.055 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:*** Caught signal (Aborted) ** 2023-11-24T09:07:53.055 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: in thread 7fceae979640 thread_name:bstore_aio 2023-11-24T09:07:53.056 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev) 2023-11-24T09:07:53.056 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcebc770520] 2023-11-24T09:07:53.056 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2: pthread_kill() 2023-11-24T09:07:53.056 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 3: raise() 2023-11-24T09:07:53.056 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 4: abort() 2023-11-24T09:07:53.056 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55ea6f134585] 2023-11-24T09:07:53.056 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x55ea6fa10fac] 2023-11-24T09:07:53.056 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 7: ceph-osd(+0x15b3861) [0x55ea6fa11861] 2023-11-24T09:07:53.056 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcebc7c2b43] 2023-11-24T09:07:53.057 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcebc854a00] 2023-11-24T09:07:53.057 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T09:07:53.053+0000 7fcea716a640 -1 bdev(0x55ea73b46e00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-11-24T09:07:53.057 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T09:07:53.053+0000 7fceae979640 -1 *** Caught signal (Aborted) ** 2023-11-24T09:07:53.057 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: in thread 7fceae979640 thread_name:bstore_aio 2023-11-24T09:07:53.057 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.057 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev) 2023-11-24T09:07:53.057 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcebc770520] 2023-11-24T09:07:53.057 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2: pthread_kill() 2023-11-24T09:07:53.058 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 3: raise() 2023-11-24T09:07:53.058 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 4: abort() 2023-11-24T09:07:53.058 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55ea6f134585] 2023-11-24T09:07:53.058 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x55ea6fa10fac] 2023-11-24T09:07:53.058 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 7: ceph-osd(+0x15b3861) [0x55ea6fa11861] 2023-11-24T09:07:53.058 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcebc7c2b43] 2023-11-24T09:07:53.058 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcebc854a00] 2023-11-24T09:07:53.058 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-24T09:07:53.058 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.072 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: -490> 2023-11-24T09:07:53.049+0000 7fceae979640 -1 bdev(0x55ea72822e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-11-24T09:07:53.072 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: -319> 2023-11-24T09:07:53.049+0000 7fceae979640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fceae979640 time 2023-11-24T09:07:53.051612+0000 2023-11-24T09:07:53.073 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-11-24T09:07:53.073 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.073 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev) 2023-11-24T09:07:53.073 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55ea6f1344d0] 2023-11-24T09:07:53.073 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x55ea6fa10fac] 2023-11-24T09:07:53.073 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 3: ceph-osd(+0x15b3861) [0x55ea6fa11861] 2023-11-24T09:07:53.073 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcebc7c2b43] 2023-11-24T09:07:53.073 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcebc854a00] 2023-11-24T09:07:53.074 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.074 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: -47> 2023-11-24T09:07:53.053+0000 7fcea716a640 -1 bdev(0x55ea73b46e00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-11-24T09:07:53.074 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 0> 2023-11-24T09:07:53.053+0000 7fceae979640 -1 *** Caught signal (Aborted) ** 2023-11-24T09:07:53.074 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: in thread 7fceae979640 thread_name:bstore_aio 2023-11-24T09:07:53.074 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.074 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev) 2023-11-24T09:07:53.074 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcebc770520] 2023-11-24T09:07:53.075 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2: pthread_kill() 2023-11-24T09:07:53.075 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 3: raise() 2023-11-24T09:07:53.075 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 4: abort() 2023-11-24T09:07:53.075 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55ea6f134585] 2023-11-24T09:07:53.075 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x55ea6fa10fac] 2023-11-24T09:07:53.075 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 7: ceph-osd(+0x15b3861) [0x55ea6fa11861] 2023-11-24T09:07:53.075 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcebc7c2b43] 2023-11-24T09:07:53.076 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcebc854a00] 2023-11-24T09:07:53.076 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-24T09:07:53.076 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.076 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T09:07:53.053+0000 7fcea716a640 -1 bdev(0x55ea73b46e00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-11-24T09:07:53.076 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:2023-11-24T09:07:53.061+0000 7fcea716a640 -1 bdev(0x55ea73b46e00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-11-24T09:07:53.081 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: -1347> 2023-11-24T09:07:53.049+0000 7fceae979640 -1 bdev(0x55ea72822e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-11-24T09:07:53.081 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: -1176> 2023-11-24T09:07:53.049+0000 7fceae979640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fceae979640 time 2023-11-24T09:07:53.051612+0000 2023-11-24T09:07:53.081 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-11-24T09:07:53.081 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.081 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev) 2023-11-24T09:07:53.081 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55ea6f1344d0] 2023-11-24T09:07:53.081 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x55ea6fa10fac] 2023-11-24T09:07:53.081 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 3: ceph-osd(+0x15b3861) [0x55ea6fa11861] 2023-11-24T09:07:53.081 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcebc7c2b43] 2023-11-24T09:07:53.082 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcebc854a00] 2023-11-24T09:07:53.082 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.082 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: -904> 2023-11-24T09:07:53.053+0000 7fcea716a640 -1 bdev(0x55ea73b46e00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-11-24T09:07:53.082 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: -857> 2023-11-24T09:07:53.053+0000 7fceae979640 -1 *** Caught signal (Aborted) ** 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: in thread 7fceae979640 thread_name:bstore_aio 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev) 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcebc770520] 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2: pthread_kill() 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 3: raise() 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 4: abort() 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55ea6f134585] 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x55ea6fa10fac] 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 7: ceph-osd(+0x15b3861) [0x55ea6fa11861] 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fcebc7c2b43] 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fcebc854a00] 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-24T09:07:53.083 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: 2023-11-24T09:07:53.084 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: -732> 2023-11-24T09:07:53.053+0000 7fcea716a640 -1 bdev(0x55ea73b46e00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-11-24T09:07:53.084 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr: -218> 2023-11-24T09:07:53.061+0000 7fcea716a640 -1 bdev(0x55ea73b46e00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-11-24T09:07:53.223 INFO:tasks.ceph.cluster2.osd.0.smithi164.stderr:daemon-helper: command failed with exit status 5 2023-11-24T09:07:54.822 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:07:54.821+0000 7fb5ce193640 -1 rbd::mirror::ImageReplayer: 0x558a5391a000 [3/1ef70dba-0221-48db-a7a4-aef3a0f822d5] start: already running 2023-11-24T09:07:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 103062528 bytes 2023-11-24T09:07:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 136943616 bytes 2023-11-24T09:07:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 232704000 bytes 2023-11-24T09:07:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 296565760 bytes 2023-11-24T09:07:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 309627904 bytes 2023-11-24T09:07:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 344200192 bytes 2023-11-24T09:07:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 344818688 bytes 2023-11-24T09:07:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 345432064 bytes 2023-11-24T09:07:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 376761344 bytes 2023-11-24T09:07:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 423782400 bytes 2023-11-24T09:07:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 479548416 bytes 2023-11-24T09:07:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 493682688 bytes 2023-11-24T09:07:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 530758656 bytes 2023-11-24T09:07:56.746 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 568817664 bytes 2023-11-24T09:07:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 588610560 bytes 2023-11-24T09:07:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 645984256 bytes 2023-11-24T09:07:57.166 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-11-24T09:07:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 678273024 bytes 2023-11-24T09:07:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 729212928 bytes 2023-11-24T09:07:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 744829952 bytes 2023-11-24T09:07:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 782487552 bytes 2023-11-24T09:07:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 816353280 bytes 2023-11-24T09:07:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 877702144 bytes 2023-11-24T09:07:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 879327232 bytes 2023-11-24T09:07:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 900293632 bytes 2023-11-24T09:07:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 938080256 bytes 2023-11-24T09:07:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 982257664 bytes 2023-11-24T09:07:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1020158976 bytes 2023-11-24T09:07:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1060752384 bytes 2023-11-24T09:07:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1112902656 bytes 2023-11-24T09:07:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:07:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:07:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:07:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:07:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-11-24T09:07:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_7 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:07:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-11-24T09:07:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:07:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-24T09:07:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:07:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:07:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_7 present 2023-11-24T09:07:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:07:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:07:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:07:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:07:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:07:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_7 2023-11-24T09:07:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:07:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:07:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:07:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-11-24T09:07:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:07:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-24T09:07:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:07:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:07:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = deleted 2023-11-24T09:07:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:08:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_7 present 2023-11-24T09:08:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:08:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:08:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:08:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:08:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_7 2023-11-24T09:08:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-11-24T09:08:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:08:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-24T09:08:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:08:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:08:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = deleted 2023-11-24T09:08:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T09:08:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_7 present 2023-11-24T09:08:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:08:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:08:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:08:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:08:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_7 2023-11-24T09:08:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-11-24T09:08:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:08:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=16845b3230c7 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 16845b3230c7 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z -o = 16845b3230c7 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_7 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-11-24T09:08:02.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-11-24T09:08:07.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-11-24T09:08:13.378 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-11-24T09:08:18.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-11-24T09:08:20.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:08:20.785+0000 7fb5d419f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558a537961a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:08:20.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:08:20.785+0000 7fb5db1ad640 -1 rbd::mirror::ImageReplayer: 0x558a5391a000 [3/1ef70dba-0221-48db-a7a4-aef3a0f822d5] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:08:24.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-11-24T09:08:29.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-11-24T09:08:29.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:08:29.948+0000 7f8c604d1640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d547849400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-11-24T09:08:29.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:08:29.948+0000 7f8c604d1640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d547849400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d8222b82-a76f-4f5f-904a-6a59fe53ff7b, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-11-24T09:08:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:08:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-11-24T09:08:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:08:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:08:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:08:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:08:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_7 2023-11-24T09:08:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:08:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: info mirror/image_7 2023-11-24T09:08:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-24T09:08:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:08:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:08:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_7 deleted 2023-11-24T09:08:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:08:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:08:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:08:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:08:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_7 2023-11-24T09:08:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-11-24T09:08:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:08:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-24T09:08:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:08:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:08:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:08:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:08:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_7 2023-11-24T09:08:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:08:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-11-24T09:08:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_7 2023-11-24T09:08:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:08:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:08:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_7 2023-11-24T09:08:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:08:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_7 2023-11-24T09:08:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-11-24T09:08:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-11-24T09:08:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:08:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:08:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_8 2023-11-24T09:08:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_8 512M 2023-11-24T09:08:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:08:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:08:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:08:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:08:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:08:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:08:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:08:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:08:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:08:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2023-11-24T09:08:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_8 2023-11-24T09:08:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:08:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:08:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:08:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-11-24T09:08:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:08:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-11-24T09:08:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_8': 2023-11-24T09:08:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:08:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:08:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:08:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 196558c67a39 2023-11-24T09:08:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.196558c67a39 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:08:30 2023 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:08:30 2023 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:08:30 2023 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:08:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 6cb41ab6-213b-4eaf-aa2b-bce6f4df326c 2023-11-24T09:08:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:08:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_8 2023-11-24T09:08:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:08:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:08:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:08:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=7 2023-11-24T09:08:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:08:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:08:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 41218048 bytes 2023-11-24T09:08:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 58435584 bytes 2023-11-24T09:08:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 95775744 bytes 2023-11-24T09:08:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 152788992 bytes 2023-11-24T09:08:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 153414656 bytes 2023-11-24T09:08:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 185123840 bytes 2023-11-24T09:08:32.022 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 220264448 bytes 2023-11-24T09:08:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 234375168 bytes 2023-11-24T09:08:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 234988544 bytes 2023-11-24T09:08:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 242715648 bytes 2023-11-24T09:08:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 310252544 bytes 2023-11-24T09:08:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 316448768 bytes 2023-11-24T09:08:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 333845504 bytes 2023-11-24T09:08:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 334472192 bytes 2023-11-24T09:08:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 383310848 bytes 2023-11-24T09:08:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 442610688 bytes 2023-11-24T09:08:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 491185152 bytes 2023-11-24T09:08:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 563367936 bytes 2023-11-24T09:08:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 586907648 bytes 2023-11-24T09:08:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 646337536 bytes 2023-11-24T09:08:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 696151040 bytes 2023-11-24T09:08:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 734111744 bytes 2023-11-24T09:08:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 754870272 bytes 2023-11-24T09:08:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 824963072 bytes 2023-11-24T09:08:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 878627840 bytes 2023-11-24T09:08:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 934816768 bytes 2023-11-24T09:08:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 985607168 bytes 2023-11-24T09:08:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1055247360 bytes 2023-11-24T09:08:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1074232320 bytes 2023-11-24T09:08:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1092679680 bytes 2023-11-24T09:08:34.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-11-24T09:08:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1128878080 bytes 2023-11-24T09:08:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1155491840 bytes 2023-11-24T09:08:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1191597056 bytes 2023-11-24T09:08:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1283491840 bytes 2023-11-24T09:08:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1307108352 bytes 2023-11-24T09:08:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1330872320 bytes 2023-11-24T09:08:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1374945280 bytes 2023-11-24T09:08:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1483371520 bytes 2023-11-24T09:08:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1542939648 bytes 2023-11-24T09:08:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1593945088 bytes 2023-11-24T09:08:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1691909120 bytes 2023-11-24T09:08:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1715439616 bytes 2023-11-24T09:08:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1719045120 bytes 2023-11-24T09:08:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1737890816 bytes 2023-11-24T09:08:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1804419072 bytes 2023-11-24T09:08:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 1864603648 bytes 2023-11-24T09:08:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 1865231360 bytes 2023-11-24T09:08:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 1892137984 bytes 2023-11-24T09:08:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 1926837248 bytes 2023-11-24T09:08:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 1975490560 bytes 2023-11-24T09:08:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 1988812800 bytes 2023-11-24T09:08:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2022962176 bytes 2023-11-24T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-11-24T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_8 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-11-24T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:08:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=17d72cd8d3d9 2023-11-24T09:08:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:08:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_8 present 17d72cd8d3d9 2023-11-24T09:08:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17d72cd8d3d9 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_8 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-11-24T09:08:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=17d72cd8d3d9 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 17d72cd8d3d9 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 17d72cd8d3d9 -o 17d72cd8d3d9 = 17d72cd8d3d9 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_8 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:08:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:08:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-11-24T09:08:40.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-11-24T09:08:45.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-11-24T09:08:49.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:08:49.992+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad94820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:08:49.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:08:49.992+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/6cb41ab6-213b-4eaf-aa2b-bce6f4df326c] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:08:51.206 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-11-24T09:08:56.610 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-11-24T09:08:56.961 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.25393651362412983) > weight (0.25) 2023-11-24T09:08:56.961 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.591901932782765) > weight (0.25) 2023-11-24T09:08:56.961 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4065656825032966) > weight (0.25) 2023-11-24T09:08:56.961 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-24T09:08:56.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-24T09:08:56.962 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-11-24T09:08:56.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:08:56.960+0000 7fb5db9ae640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 56194) UID: 1000 2023-11-24T09:08:58.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:08:58.052+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad94ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:08:58.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:08:58.052+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/6cb41ab6-213b-4eaf-aa2b-bce6f4df326c] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:09:00.962 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-24T09:09:00.963 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:09:00.963 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-24T09:09:00.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-24T09:09:00.963 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-24T09:09:00.966 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-24T09:09:00.966 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-11-24T09:09:02.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-11-24T09:09:07.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-11-24T09:09:07.673 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.994992233580699) > weight (0.25) 2023-11-24T09:09:07.673 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-24T09:09:07.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-24T09:09:07.674 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-24T09:09:07.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:09:07.672+0000 7f2661999640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 74558) UID: 1000 2023-11-24T09:09:08.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:09:08.868+0000 7f024411a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959ad88000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:09:08.873 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:09:08.868+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/6cb41ab6-213b-4eaf-aa2b-bce6f4df326c] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:09:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:09:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-11-24T09:09:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:09:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:09:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:09:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_8 2023-11-24T09:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-11-24T09:09:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:09:09.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-11-24T09:09:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:09:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:09:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_8 deleted 2023-11-24T09:09:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:09:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:09:09.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:09:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:09:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:09:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_8 2023-11-24T09:09:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:09:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-11-24T09:09:09.191 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:09:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-11-24T09:09:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_8 2023-11-24T09:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:09:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2023-11-24T09:09:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_8 2023-11-24T09:09:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:09:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:09:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_8 2023-11-24T09:09:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_8 2023-11-24T09:09:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-11-24T09:09:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-11-24T09:09:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:09:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_9 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_9 512M 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:09:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:09:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:09:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:09:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2023-11-24T09:09:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_9 2023-11-24T09:09:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:09:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-11-24T09:09:09.674 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-24T09:09:09.675 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:09:09.675 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-24T09:09:09.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-24T09:09:09.675 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-24T09:09:09.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-24T09:09:09.677 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-11-24T09:09:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:09:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-11-24T09:09:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_9': 2023-11-24T09:09:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:09:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:09:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:09:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 19842b23faf6 2023-11-24T09:09:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.19842b23faf6 2023-11-24T09:09:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:09:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:09:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:09:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:09:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:09:09 2023 2023-11-24T09:09:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:09:09 2023 2023-11-24T09:09:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:09:09 2023 2023-11-24T09:09:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:09:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:09:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 127932ea-757b-4389-b598-95229c278033 2023-11-24T09:09:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:09:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_9 2023-11-24T09:09:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:09:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=10 2023-11-24T09:09:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:09:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:09:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 31761408 bytes 2023-11-24T09:09:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 68825088 bytes 2023-11-24T09:09:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 91271168 bytes 2023-11-24T09:09:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 188169216 bytes 2023-11-24T09:09:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 201587712 bytes 2023-11-24T09:09:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 271038464 bytes 2023-11-24T09:09:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 335134720 bytes 2023-11-24T09:09:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 349149184 bytes 2023-11-24T09:09:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 421763072 bytes 2023-11-24T09:09:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 527987712 bytes 2023-11-24T09:09:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 528583680 bytes 2023-11-24T09:09:12.104 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:09:12.100+0000 7f42b0626640 -1 reset not still connected to 0x561a4e773860 2023-11-24T09:09:12.105 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:09:12.100+0000 7f42b0626640 -1 reset not still connected to 0x561a513eec30 2023-11-24T09:09:12.105 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:09:12.100+0000 7f42b0626640 -1 reset not still connected to 0x561a5af1dee0 2023-11-24T09:09:12.106 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:09:12.100+0000 7f42b0626640 -1 reset not still connected to 0x561a79b74410 2023-11-24T09:09:12.106 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:09:12.100+0000 7f42b0626640 -1 reset not still connected to 0x561a8349e270 2023-11-24T09:09:12.106 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:09:12.100+0000 7f42b0626640 -1 reset not still connected to 0x561a912ccf70 2023-11-24T09:09:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 559009792 bytes 2023-11-24T09:09:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 589127680 bytes 2023-11-24T09:09:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 621124608 bytes 2023-11-24T09:09:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 621742080 bytes 2023-11-24T09:09:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 690794496 bytes 2023-11-24T09:09:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 722817024 bytes 2023-11-24T09:09:12.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-11-24T09:09:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 792854528 bytes 2023-11-24T09:09:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 807410688 bytes 2023-11-24T09:09:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 843928576 bytes 2023-11-24T09:09:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 944964608 bytes 2023-11-24T09:09:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 975360000 bytes 2023-11-24T09:09:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 975983616 bytes 2023-11-24T09:09:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 985620480 bytes 2023-11-24T09:09:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1021496320 bytes 2023-11-24T09:09:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1022117888 bytes 2023-11-24T09:09:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1132938240 bytes 2023-11-24T09:09:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1174181888 bytes 2023-11-24T09:09:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1211829248 bytes 2023-11-24T09:09:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1279097856 bytes 2023-11-24T09:09:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1305937920 bytes 2023-11-24T09:09:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1375015936 bytes 2023-11-24T09:09:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1404358656 bytes 2023-11-24T09:09:15.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1475057664 bytes 2023-11-24T09:09:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1530868736 bytes 2023-11-24T09:09:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1570330624 bytes 2023-11-24T09:09:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1634094080 bytes 2023-11-24T09:09:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1696198656 bytes 2023-11-24T09:09:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1765813248 bytes 2023-11-24T09:09:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1804606464 bytes 2023-11-24T09:09:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 1821114368 bytes 2023-11-24T09:09:16.294 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 1825141760 bytes 2023-11-24T09:09:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 1859811328 bytes 2023-11-24T09:09:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 1912421376 bytes 2023-11-24T09:09:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 1964921856 bytes 2023-11-24T09:09:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2009190400 bytes 2023-11-24T09:09:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2036109312 bytes 2023-11-24T09:09:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2040771584 bytes 2023-11-24T09:09:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2045927424 bytes 2023-11-24T09:09:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2063641600 bytes 2023-11-24T09:09:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2093121536 bytes 2023-11-24T09:09:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2117772288 bytes 2023-11-24T09:09:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2197214208 bytes 2023-11-24T09:09:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2230117376 bytes 2023-11-24T09:09:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2249071616 bytes 2023-11-24T09:09:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2281313280 bytes 2023-11-24T09:09:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2381968384 bytes 2023-11-24T09:09:18.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-11-24T09:09:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2417126400 bytes 2023-11-24T09:09:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2469472256 bytes 2023-11-24T09:09:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2493301760 bytes 2023-11-24T09:09:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2524032000 bytes 2023-11-24T09:09:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2571147264 bytes 2023-11-24T09:09:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2648057856 bytes 2023-11-24T09:09:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2675349504 bytes 2023-11-24T09:09:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2688079872 bytes 2023-11-24T09:09:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2728856576 bytes 2023-11-24T09:09:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2756950016 bytes 2023-11-24T09:09:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2784017408 bytes 2023-11-24T09:09:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 2816891904 bytes 2023-11-24T09:09:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 2878292992 bytes 2023-11-24T09:09:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:09:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:09:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_9 2023-11-24T09:09:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-11-24T09:09:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:09:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=1684f3d5b649 2023-11-24T09:09:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:09:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_9 present 1684f3d5b649 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=1684f3d5b649 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_9 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-11-24T09:09:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:09:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=1684f3d5b649 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1684f3d5b649 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 1684f3d5b649 -o 1684f3d5b649 = 1684f3d5b649 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_9 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2023-11-24T09:09:23.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-11-24T09:09:29.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-11-24T09:09:31.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:09:31.055+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959aedf520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:09:31.305 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:09:31.055+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/127932ea-757b-4389-b598-95229c278033] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:09:36.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~100s 2023-11-24T09:09:42.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~105s 2023-11-24T09:09:47.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~110s 2023-11-24T09:09:49.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:09:49.991+0000 7f0243919640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55959aedfd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:09:49.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:09:49.991+0000 7f024a927640 -1 rbd::mirror::ImageReplayer: 0x55959ab5a280 [3/127932ea-757b-4389-b598-95229c278033] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:09:50.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:09:50.975+0000 7f8c604d1640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d54788b800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2023-11-24T09:09:50.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:09:50.975+0000 7f8c604d1640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d54788b800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=388, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d8222b82-a76f-4f5f-904a-6a59fe53ff7b, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2023-11-24T09:09:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:09:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-11-24T09:09:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:09:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:09:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:09:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:09:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_9 2023-11-24T09:09:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:09:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd --cluster=cluster1 info mirror/image_9 2023-11-24T09:09:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-11-24T09:09:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id= 2023-11-24T09:09:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:09:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_9 deleted 2023-11-24T09:09:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:09:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:09:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:09:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:09:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_9 2023-11-24T09:09:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:09:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-11-24T09:09:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:09:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-11-24T09:09:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:09:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:09:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:09:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:09:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_9 2023-11-24T09:09:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2023-11-24T09:09:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_9 2023-11-24T09:09:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:09:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_9 2023-11-24T09:09:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_9 2023-11-24T09:09:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-11-24T09:09:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-11-24T09:09:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image=image_10 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ create_image cluster2 mirror image_10 512M 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local size=128 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -n 512M ] 2023-11-24T09:09:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ size=512M 2023-11-24T09:09:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ shift 2023-11-24T09:09:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2023-11-24T09:09:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ enable_mirror cluster2 mirror image_10 2023-11-24T09:09:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:09:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local mode=snapshot 2023-11-24T09:09:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-11-24T09:09:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring enabled 2023-11-24T09:09:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-11-24T09:09:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:rbd image 'image_10': 2023-11-24T09:09:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: size 512 MiB in 128 objects 2023-11-24T09:09:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: order 22 (4 MiB objects) 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: snapshot_count: 1 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: id: 19a21b03a1e6 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: block_name_prefix: rbd_data.19a21b03a1e6 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: format: 2 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: features: layering, exclusive-lock, object-map, fast-diff 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: op_features: 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: flags: 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: create_timestamp: Fri Nov 24 09:09:51 2023 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: access_timestamp: Fri Nov 24 09:09:51 2023 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: modify_timestamp: Fri Nov 24 09:09:51 2023 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring state: enabled 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring mode: snapshot 2023-11-24T09:09:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring global id: 96961940-3d16-41ed-abfc-d9e6df688f9e 2023-11-24T09:09:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout: mirroring primary: true 2023-11-24T09:09:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stress_write_image cluster2 mirror image_10 2023-11-24T09:09:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:09:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:09:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:09:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-11-24T09:09:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local duration=11 2023-11-24T09:09:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:09:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-11-24T09:09:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:100 IOs, 2516992 bytes 2023-11-24T09:09:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:200 IOs, 72115200 bytes 2023-11-24T09:09:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:300 IOs, 107233280 bytes 2023-11-24T09:09:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:400 IOs, 133608448 bytes 2023-11-24T09:09:52.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~116s 2023-11-24T09:09:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:500 IOs, 192688128 bytes 2023-11-24T09:09:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:600 IOs, 253187072 bytes 2023-11-24T09:09:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:700 IOs, 352128000 bytes 2023-11-24T09:09:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:800 IOs, 423230464 bytes 2023-11-24T09:09:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:900 IOs, 465927168 bytes 2023-11-24T09:09:53.773 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1000 IOs, 499149824 bytes 2023-11-24T09:09:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1100 IOs, 549781504 bytes 2023-11-24T09:09:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1200 IOs, 601736192 bytes 2023-11-24T09:09:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1300 IOs, 693855232 bytes 2023-11-24T09:09:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1400 IOs, 711341056 bytes 2023-11-24T09:09:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1500 IOs, 748615680 bytes 2023-11-24T09:09:54.624 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1600 IOs, 787718144 bytes 2023-11-24T09:09:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1700 IOs, 820669440 bytes 2023-11-24T09:09:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1800 IOs, 880532480 bytes 2023-11-24T09:09:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:1900 IOs, 935207936 bytes 2023-11-24T09:09:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2000 IOs, 995483648 bytes 2023-11-24T09:09:55.319 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2100 IOs, 1046026240 bytes 2023-11-24T09:09:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2200 IOs, 1101323264 bytes 2023-11-24T09:09:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2300 IOs, 1178774528 bytes 2023-11-24T09:09:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2400 IOs, 1226283008 bytes 2023-11-24T09:09:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2500 IOs, 1250936832 bytes 2023-11-24T09:09:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2600 IOs, 1342920704 bytes 2023-11-24T09:09:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2700 IOs, 1370541056 bytes 2023-11-24T09:09:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2800 IOs, 1405007872 bytes 2023-11-24T09:09:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:2900 IOs, 1481891840 bytes 2023-11-24T09:09:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3000 IOs, 1542742016 bytes 2023-11-24T09:09:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3100 IOs, 1589083136 bytes 2023-11-24T09:09:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3200 IOs, 1630876672 bytes 2023-11-24T09:09:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3300 IOs, 1631481856 bytes 2023-11-24T09:09:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3400 IOs, 1658806272 bytes 2023-11-24T09:09:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3500 IOs, 1721462784 bytes 2023-11-24T09:09:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3600 IOs, 1815973888 bytes 2023-11-24T09:09:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3700 IOs, 1875993600 bytes 2023-11-24T09:09:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3800 IOs, 1915157504 bytes 2023-11-24T09:09:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:3900 IOs, 1988781056 bytes 2023-11-24T09:09:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4000 IOs, 1989384192 bytes 2023-11-24T09:09:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4100 IOs, 2027290624 bytes 2023-11-24T09:09:58.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~121s 2023-11-24T09:09:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4200 IOs, 2064774144 bytes 2023-11-24T09:09:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4300 IOs, 2118643712 bytes 2023-11-24T09:09:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4400 IOs, 2163168256 bytes 2023-11-24T09:09:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4500 IOs, 2197675008 bytes 2023-11-24T09:09:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4600 IOs, 2218366976 bytes 2023-11-24T09:09:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4700 IOs, 2242157568 bytes 2023-11-24T09:09:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4800 IOs, 2313110528 bytes 2023-11-24T09:09:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:4900 IOs, 2331388928 bytes 2023-11-24T09:09:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5000 IOs, 2373951488 bytes 2023-11-24T09:09:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5100 IOs, 2418246656 bytes 2023-11-24T09:09:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5200 IOs, 2438102016 bytes 2023-11-24T09:09:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5300 IOs, 2491423744 bytes 2023-11-24T09:09:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5400 IOs, 2520055808 bytes 2023-11-24T09:10:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5500 IOs, 2540015616 bytes 2023-11-24T09:10:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5600 IOs, 2591978496 bytes 2023-11-24T09:10:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5700 IOs, 2614295552 bytes 2023-11-24T09:10:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5800 IOs, 2637309952 bytes 2023-11-24T09:10:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:5900 IOs, 2684410880 bytes 2023-11-24T09:10:00.793 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6000 IOs, 2747962368 bytes 2023-11-24T09:10:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6100 IOs, 2806214656 bytes 2023-11-24T09:10:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6200 IOs, 2817509376 bytes 2023-11-24T09:10:01.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6300 IOs, 2881076224 bytes 2023-11-24T09:10:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6400 IOs, 2918778880 bytes 2023-11-24T09:10:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6500 IOs, 2940208128 bytes 2023-11-24T09:10:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6600 IOs, 2955427840 bytes 2023-11-24T09:10:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6700 IOs, 2987911168 bytes 2023-11-24T09:10:01.781 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6800 IOs, 2988511232 bytes 2023-11-24T09:10:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:6900 IOs, 3037027328 bytes 2023-11-24T09:10:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7000 IOs, 3143463936 bytes 2023-11-24T09:10:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7100 IOs, 3207387136 bytes 2023-11-24T09:10:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7200 IOs, 3251891200 bytes 2023-11-24T09:10:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7300 IOs, 3271144448 bytes 2023-11-24T09:10:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7400 IOs, 3363035136 bytes 2023-11-24T09:10:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7500 IOs, 3393840128 bytes 2023-11-24T09:10:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:7600 IOs, 3399965696 bytes 2023-11-24T09:10:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=124 2023-11-24T09:10:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:10:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 124 -eq 124 ] 2023-11-24T09:10:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=present 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:10:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:10:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=17d78a86982d 2023-11-24T09:10:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:10:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 present 17d78a86982d 2023-11-24T09:10:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:03.252 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=present 2023-11-24T09:10:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17d78a86982d 2023-11-24T09:10:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:10:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:10:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:10:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:10:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:10:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=17d78a86982d 2023-11-24T09:10:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 17d78a86982d 2023-11-24T09:10:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 17d78a86982d -o 17d78a86982d = 17d78a86982d 2023-11-24T09:10:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:10:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test present = present 2023-11-24T09:10:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:10:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ disable_mirror cluster2 mirror image_10 2023-11-24T09:10:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:10:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2023-11-24T09:10:03.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~127s 2023-11-24T09:10:09.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~132s 2023-11-24T09:10:14.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~137s 2023-11-24T09:10:19.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~143s 2023-11-24T09:10:25.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~148s 2023-11-24T09:10:26.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:10:26.230+0000 7fd257121640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a209c456c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-24T09:10:26.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:10:26.230+0000 7fd25d92e640 -1 rbd::mirror::ImageReplayer: 0x55a2090fa280 [3/96961940-3d16-41ed-abfc-d9e6df688f9e] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-24T09:10:30.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~154s 2023-11-24T09:10:32.678 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-24T09:10:32.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-24T09:10:32.679 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-24T09:10:32.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:10:32.678+0000 7f09fe613640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 73742) UID: 1000 2023-11-24T09:10:33.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:10:33.822+0000 7f8c6d4eb640 -1 rbd::mirror::ImageMap: 0x55d54769dc20 handle_peer_ack_remove: failed to remove global_image_id=96961940-3d16-41ed-abfc-d9e6df688f9e 2023-11-24T09:10:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:Mirroring disabled 2023-11-24T09:10:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-11-24T09:10:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local state=deleted 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id= 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local s 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:10:33.869 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:10:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ image_id=17d78a86982d 2023-11-24T09:10:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0.1 2023-11-24T09:10:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted 17d78a86982d 2023-11-24T09:10:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:10:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17d78a86982d 2023-11-24T09:10:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:10:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:10:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:10:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:10:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:10:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=17d78a86982d 2023-11-24T09:10:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 17d78a86982d 2023-11-24T09:10:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 17d78a86982d -o 17d78a86982d = 17d78a86982d 2023-11-24T09:10:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:10:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:10:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 1 2023-11-24T09:10:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted 17d78a86982d 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17d78a86982d 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:10:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:10:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=17d78a86982d 2023-11-24T09:10:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 17d78a86982d 2023-11-24T09:10:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 17d78a86982d -o 17d78a86982d = 17d78a86982d 2023-11-24T09:10:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:10:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:10:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 2 2023-11-24T09:10:35.679 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-24T09:10:35.680 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:10:35.680 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-24T09:10:35.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-24T09:10:35.680 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-24T09:10:35.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-24T09:10:35.683 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-11-24T09:10:36.151 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~159s 2023-11-24T09:10:36.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.29047646928710724) > weight (0.25) 2023-11-24T09:10:36.968 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4188735456294749) > weight (0.25) 2023-11-24T09:10:36.968 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-11-24T09:10:36.968 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-11-24T09:10:36.968 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-11-24T09:10:36.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:10:36.966+0000 7f024b128640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 63609) UID: 1000 2023-11-24T09:10:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted 17d78a86982d 2023-11-24T09:10:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:10:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17d78a86982d 2023-11-24T09:10:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:10:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:10:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:10:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:10:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:10:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=17d78a86982d 2023-11-24T09:10:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 17d78a86982d 2023-11-24T09:10:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 17d78a86982d -o 17d78a86982d = 17d78a86982d 2023-11-24T09:10:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:10:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:10:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 4 2023-11-24T09:10:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted 17d78a86982d 2023-11-24T09:10:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:10:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17d78a86982d 2023-11-24T09:10:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:10:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:10:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:10:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:41.232 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:10:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:10:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=17d78a86982d 2023-11-24T09:10:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 17d78a86982d 2023-11-24T09:10:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 17d78a86982d -o 17d78a86982d = 17d78a86982d 2023-11-24T09:10:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:10:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:10:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T09:10:41.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~164s 2023-11-24T09:10:45.969 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-11-24T09:10:45.969 INFO:teuthology.orchestra.run:waiting for 600 2023-11-24T09:10:45.969 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-11-24T09:10:45.969 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-11-24T09:10:45.970 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-11-24T09:10:45.972 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-11-24T09:10:45.972 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-11-24T09:10:46.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~170s 2023-11-24T09:10:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted 17d78a86982d 2023-11-24T09:10:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17d78a86982d 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:10:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=17d78a86982d 2023-11-24T09:10:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 17d78a86982d 2023-11-24T09:10:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 17d78a86982d -o 17d78a86982d = 17d78a86982d 2023-11-24T09:10:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:10:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:10:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T09:10:52.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~175s 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted 17d78a86982d 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17d78a86982d 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:10:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:10:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:10:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:10:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:10:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id=17d78a86982d 2023-11-24T09:10:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 17d78a86982d 2023-11-24T09:10:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -z 17d78a86982d -o 17d78a86982d = 17d78a86982d 2023-11-24T09:10:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_state=present 2023-11-24T09:10:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = present 2023-11-24T09:10:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 8 2023-11-24T09:10:57.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~181s 2023-11-24T09:11:03.170 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~186s 2023-11-24T09:11:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test_image_present cluster1 mirror image_10 deleted 17d78a86982d 2023-11-24T09:11:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local test_state=deleted 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image_id=17d78a86982d 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_state=deleted 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local current_image_id 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ get_image_id cluster1 mirror image_10 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-11-24T09:11:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-11-24T09:11:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-11-24T09:11:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ current_image_id= 2023-11-24T09:11:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 2023-11-24T09:11:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test deleted = deleted 2023-11-24T09:11:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:11:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ purge_snapshots cluster2 mirror image_10 2023-11-24T09:11:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:11:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2023-11-24T09:11:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image_retry cluster2 mirror image_10 2023-11-24T09:11:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:11:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:11:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ remove_image cluster2 mirror image_10 2023-11-24T09:11:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local image=image_10 2023-11-24T09:11:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-11-24T09:11:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-11-24T09:11:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-24T09:11:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:11:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ testlog TEST: check if removed images' OMAP are removed 2023-11-24T09:11:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + tee -a /tmp/tmp.tyy3Nh9UuT/rbd-mirror.test.log 2023-11-24T09:11:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:date +%F %T 2023-11-24T09:11:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo 2023-11-24 09:11:05 TEST: check if removed images' OMAP are removed 2023-11-24T09:11:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:2023-11-24 09:11:05 TEST: check if removed images' OMAP are removed 2023-11-24T09:11:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_in_omap cluster1 mirror 2023-11-24T09:11:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=status_global 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=status_global 2023-11-24T09:11:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ list_omap_keys cluster1 mirror rbd_mirroring 2023-11-24T09:11:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ + rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-11-24T09:11:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:grep -c status_global 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 0 ] 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_ 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_ 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ list_omap_keys cluster1 mirror rbd_mirroring 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-11-24T09:11:05.959 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -c image_ 2023-11-24T09:11:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2023-11-24T09:11:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T09:11:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:11:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 0 ] 2023-11-24T09:11:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirror_leader 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_map 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirror_leader 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_map 2023-11-24T09:11:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-11-24T09:11:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirror_leader 2023-11-24T09:11:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-11-24T09:11:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -c image_map 2023-11-24T09:11:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2023-11-24T09:11:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T09:11:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:11:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 0 ] 2023-11-24T09:11:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:11:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_image_in_omap cluster2 mirror 2023-11-24T09:11:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=status_global 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=status_global 2023-11-24T09:11:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ list_omap_keys cluster2 mirror rbd_mirroring 2023-11-24T09:11:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-11-24T09:11:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -c status_global 2023-11-24T09:11:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2023-11-24T09:11:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T09:11:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:11:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 0 ] 2023-11-24T09:11:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:11:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2023-11-24T09:11:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_ 2023-11-24T09:11:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:11:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:11:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2023-11-24T09:11:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_ 2023-11-24T09:11:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ list_omap_keys cluster2 mirror rbd_mirroring 2023-11-24T09:11:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirroring 2023-11-24T09:11:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-11-24T09:11:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -c image_ 2023-11-24T09:11:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 0 ] 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirror_leader 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_map 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ sleep 0 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:11:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirror_leader 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local filter=image_map 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ list_omap_keys cluster2 mirror rbd_mirror_leader 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local pool=mirror 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local obj_name=rbd_mirror_leader 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2023-11-24T09:11:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ grep -c image_map 2023-11-24T09:11:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test 0 = 0 2023-11-24T09:11:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ error_code=0 2023-11-24T09:11:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set -e 2023-11-24T09:11:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 0 ] 2023-11-24T09:11:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 0 2023-11-24T09:11:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ cleanup 0 2023-11-24T09:11:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local error_code=0 2023-11-24T09:11:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ set +e 2023-11-24T09:11:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -ne 0 ] 2023-11-24T09:11:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -z ] 2023-11-24T09:11:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster instance 2023-11-24T09:11:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-11-24T09:11:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror' does not exist 2023-11-24T09:11:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-11-24T09:11:07.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:07.866+0000 7f8c6d4eb640 -1 rbd::mirror::LeaderWatcher: 0x55d5474afb00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-11-24T09:11:07.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:07.866+0000 7f8c674df640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55d5475fbb00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-24T09:11:07.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:07.866+0000 7f8c674df640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55d54782db00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-24T09:11:07.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:07.866+0000 7f8c674df640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55d54782d440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-24T09:11:07.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:07.866+0000 7f8c66cde640 -1 rbd::mirror::PoolWatcher: 0x55d5475483c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:07.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:07.866+0000 7f8c66cde640 -1 rbd::mirror::PoolWatcher: 0x55d547548000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:07.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:07.866+0000 7f8c674df640 -1 rbd::mirror::PoolWatcher: 0x55d5458d3e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:07.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:07.874+0000 7f8c6d4eb640 -1 rbd::mirror::LeaderWatcher: 0x55d5474afb00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-11-24T09:11:07.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:07.874+0000 7f8c6d4eb640 -1 rbd::mirror::LeaderWatcher: 0x55d5474afb00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-11-24T09:11:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror' does not exist 2023-11-24T09:11:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-11-24T09:11:08.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~191s 2023-11-24T09:11:09.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.226+0000 7f87b9307640 -1 rbd::mirror::RemotePoolPoller: 0x55860fa59930 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-11-24T09:11:09.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.226+0000 7f87b9b08640 -1 rbd::mirror::RemotePoolPoller: 0x55860fa59930 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-11-24T09:11:09.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.226+0000 7f87b9b08640 -1 rbd::mirror::PoolReplayer: 0x558610928000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-11-24T09:11:09.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.226+0000 7f87c0315640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55861178eb40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-24T09:11:09.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.226+0000 7f87c0315640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558610dcb200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-24T09:11:09.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.226+0000 7f87c0315640 -1 rbd::mirror::PoolWatcher: 0x55860fa210e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:09.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.226+0000 7f87c0315640 -1 rbd::mirror::PoolWatcher: 0x55860fa21860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:09.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.226+0000 7f87b9307640 -1 rbd::mirror::PoolWatcher: 0x55860fa21c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:09.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.226+0000 7f87b9b08640 -1 rbd::mirror::PoolWatcher: 0x55860fa21e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:09.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87c6321640 -1 rbd::mirror::InstanceWatcher: 0x55861091f500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6460 object: (2) No such file or directory 2023-11-24T09:11:09.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87c6321640 -1 rbd::mirror::InstanceWatcher: 0x558610e84380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6460 object: (2) No such file or directory 2023-11-24T09:11:09.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87bfb14640 -1 rbd::mirror::InstanceWatcher: 0x55861091f500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:09.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87bfb14640 -1 rbd::mirror::InstanceWatcher: 0x558610e84380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:09.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87b9b08640 -1 rbd::mirror::MirrorStatusUpdater 0x5586109ec360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87b9b08640 -1 rbd::mirror::MirrorStatusUpdater 0x5586109ec480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87c6321640 -1 rbd::mirror::NamespaceReplayer: 0x55861099c820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87c6321640 -1 rbd::mirror::NamespaceReplayer: 0x55861099c9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87c0315640 -1 rbd::mirror::MirrorStatusUpdater 0x5586109ec000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87c0315640 -1 rbd::mirror::MirrorStatusUpdater 0x5586109ec120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87c6b22640 -1 rbd::mirror::NamespaceReplayer: 0x55861099c820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.230+0000 7f87c6b22640 -1 rbd::mirror::NamespaceReplayer: 0x55861099c9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror_parent' does not exist 2023-11-24T09:11:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-11-24T09:11:09.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.426+0000 7f87abaec640 -1 rbd::mirror::RemotePoolPoller: 0x558610d24000 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-11-24T09:11:09.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.426+0000 7f87abaec640 -1 rbd::mirror::RemotePoolPoller: 0x558610d24000 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-11-24T09:11:09.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:09.426+0000 7f87abaec640 -1 rbd::mirror::PoolReplayer: 0x5586109bc000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-11-24T09:11:09.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4f9f5bf640 -1 rbd::mirror::RemotePoolPoller: 0x5632b3df7860 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-11-24T09:11:09.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4f9edbe640 -1 rbd::mirror::RemotePoolPoller: 0x5632b3df7860 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-11-24T09:11:09.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4f9edbe640 -1 rbd::mirror::PoolReplayer: 0x5632b4cc0000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-11-24T09:11:09.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fabdd8640 -1 rbd::mirror::InstanceWatcher: 0x5632b4d34000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6248 object: (2) No such file or directory 2023-11-24T09:11:09.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fac5d9640 -1 rbd::mirror::InstanceWatcher: 0x5632b4d28700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6248 object: (2) No such file or directory 2023-11-24T09:11:09.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fa5dcc640 -1 rbd::mirror::InstanceWatcher: 0x5632b4d34000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:09.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fa55cb640 -1 rbd::mirror::InstanceWatcher: 0x5632b4d28700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:09.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4f9f5bf640 -1 rbd::mirror::MirrorStatusUpdater 0x5632b4d9e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fabdd8640 -1 rbd::mirror::NamespaceReplayer: 0x5632b4d22820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4f9edbe640 -1 rbd::mirror::MirrorStatusUpdater 0x5632b4f68a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fac5d9640 -1 rbd::mirror::NamespaceReplayer: 0x5632b4d229c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fa5dcc640 -1 rbd::mirror::MirrorStatusUpdater 0x5632b4d9e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fabdd8640 -1 rbd::mirror::NamespaceReplayer: 0x5632b4d22820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fa55cb640 -1 rbd::mirror::MirrorStatusUpdater 0x5632b4d9e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:09.858+0000 7f4fac5d9640 -1 rbd::mirror::NamespaceReplayer: 0x5632b4d229c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.954+0000 7f8c6dcec640 -1 rbd::mirror::InstanceWatcher: 0x55d546ab9180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5764 object: (2) No such file or directory 2023-11-24T09:11:09.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.954+0000 7f8c6d4eb640 -1 rbd::mirror::InstanceWatcher: 0x55d546ab9880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5764 object: (2) No such file or directory 2023-11-24T09:11:09.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.954+0000 7f8c674df640 -1 rbd::mirror::InstanceWatcher: 0x55d546ab9180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:09.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.954+0000 7f8c674df640 -1 rbd::mirror::InstanceWatcher: 0x55d546ab9880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:09.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.954+0000 7f8c60cd2640 -1 rbd::mirror::MirrorStatusUpdater 0x55d546ab0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.954+0000 7f8c6dcec640 -1 rbd::mirror::NamespaceReplayer: 0x55d546a48820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.954+0000 7f8c604d1640 -1 rbd::mirror::MirrorStatusUpdater 0x55d546ab0480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.954+0000 7f8c6d4eb640 -1 rbd::mirror::NamespaceReplayer: 0x55d546a489c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.958+0000 7f8c674df640 -1 rbd::mirror::MirrorStatusUpdater 0x55d546ab0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.958+0000 7f8c674df640 -1 rbd::mirror::MirrorStatusUpdater 0x55d546ab0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:09.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.958+0000 7f8c6dcec640 -1 rbd::mirror::NamespaceReplayer: 0x55d546a489c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:09.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:09.958+0000 7f8c6dcec640 -1 rbd::mirror::NamespaceReplayer: 0x55d546a48820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.062+0000 7f8c60cd2640 -1 rbd::mirror::RemotePoolPoller: 0x55d54590ba00 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-11-24T09:11:10.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.066+0000 7f8c60cd2640 -1 rbd::mirror::RemotePoolPoller: 0x55d54590ba00 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-11-24T09:11:10.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.066+0000 7f8c60cd2640 -1 rbd::mirror::PoolReplayer: 0x55d5467da000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-11-24T09:11:10.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.122+0000 7f4f91da4640 -1 rbd::mirror::RemotePoolPoller: 0x5632b3df7ee0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-11-24T09:11:10.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.122+0000 7f4f91da4640 -1 rbd::mirror::RemotePoolPoller: 0x5632b3df7ee0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-11-24T09:11:10.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.122+0000 7f4f91da4640 -1 rbd::mirror::PoolReplayer: 0x5632b5a78600 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-11-24T09:11:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:pool 'mirror_parent' does not exist 2023-11-24T09:11:10.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.146+0000 7f4fabdd8640 -1 rbd::mirror::LeaderWatcher: 0x5632b5a78300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:10.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.146+0000 7f4fabdd8640 -1 rbd::mirror::InstanceWatcher: 0x5632b4ca3880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6248 object: (2) No such file or directory 2023-11-24T09:11:10.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.146+0000 7f4fa55cb640 -1 rbd::mirror::InstanceWatcher: 0x5632b4ca3880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:10.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.150+0000 7f4f9f5bf640 -1 rbd::mirror::MirrorStatusUpdater 0x5632b59b06c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.150+0000 7f4fabdd8640 -1 rbd::mirror::NamespaceReplayer: 0x5632b4d22000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.150+0000 7f4fa5dcc640 -1 rbd::mirror::MirrorStatusUpdater 0x5632b5902c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.150+0000 7f4fac5d9640 -1 rbd::mirror::NamespaceReplayer: 0x5632b4d22000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirrors cluster1 2023-11-24T09:11:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1 2023-11-24T09:11:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T09:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster1:0 2023-11-24T09:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:0 2023-11-24T09:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T09:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T09:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster1:1 2023-11-24T09:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:1 2023-11-24T09:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster1:2 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:2 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster1:3 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster1:3 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T09:11:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirrors cluster2 2023-11-24T09:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2 2023-11-24T09:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ seq 0 3 2023-11-24T09:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster2:0 2023-11-24T09:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:0 2023-11-24T09:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T09:11:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster2:1 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:1 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster2:2 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:2 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ stop_mirror cluster2:3 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local cluster=cluster2:3 2023-11-24T09:11:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ local sig= 2023-11-24T09:11:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test -n 1 2023-11-24T09:11:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ return 2023-11-24T09:11:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ -z 1 ] 2023-11-24T09:11:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ test = /tmp/tmp.tyy3Nh9UuT 2023-11-24T09:11:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ rm -Rf /tmp/tmp.tyy3Nh9UuT 2023-11-24T09:11:10.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:10.154+0000 7f4f97db0640 -1 rbd::mirror::InstanceWatcher: 0x5632b4ca3180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ [ 0 -eq 0 ] 2023-11-24T09:11:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stdout:OK 2023-11-24T09:11:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ echo OK 2023-11-24T09:11:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi164.stderr:+ exit 0 2023-11-24T09:11:10.159 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-24T09:11:10.159 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-11-24T09:11:10.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.206+0000 7f8c6d4eb640 -1 rbd::mirror::LeaderWatcher: 0x55d5474afb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:10.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.206+0000 7f8c6d4eb640 -1 rbd::mirror::InstanceWatcher: 0x55d5467bd880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5764 object: (2) No such file or directory 2023-11-24T09:11:10.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.210+0000 7f8c66cde640 -1 rbd::mirror::InstanceWatcher: 0x55d5467bd880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:10.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.210+0000 7f8c604d1640 -1 rbd::mirror::MirrorStatusUpdater 0x55d5474986c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.210+0000 7f8c6d4eb640 -1 rbd::mirror::NamespaceReplayer: 0x55d546a48000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.210+0000 7f8c674df640 -1 rbd::mirror::MirrorStatusUpdater 0x55d54743cc60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.210+0000 7f8c6dcec640 -1 rbd::mirror::NamespaceReplayer: 0x55d546a48000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.237 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-11-24T09:11:10.238 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-11-24T09:11:10.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.346+0000 7f8c594c3640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55d54782d8c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-24T09:11:10.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c52cb6640 -1 rbd::mirror::RemotePoolPoller: 0x55d54590bc70 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-11-24T09:11:10.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c52cb6640 -1 rbd::mirror::PoolWatcher: 0x55d5458d3a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:10.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c52cb6640 -1 rbd::mirror::RemotePoolPoller: 0x55d54590bc70 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-11-24T09:11:10.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c52cb6640 -1 rbd::mirror::PoolReplayer: 0x55d546a56000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-11-24T09:11:10.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c6d4eb640 -1 rbd::mirror::LeaderWatcher: 0x55d546e58300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-11-24T09:11:10.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c6d4eb640 -1 rbd::mirror::LeaderWatcher: 0x55d546e58300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-11-24T09:11:10.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c6dcec640 -1 rbd::mirror::LeaderWatcher: 0x55d546e58300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:10.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c594c3640 -1 rbd::mirror::InstanceWatcher: 0x55d546acb180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:10.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c52cb6640 -1 rbd::mirror::MirrorStatusUpdater 0x55d546cc2900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c6d4eb640 -1 rbd::mirror::NamespaceReplayer: 0x55d546e5a4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c594c3640 -1 rbd::mirror::MirrorStatusUpdater 0x55d546c7ca20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:10.362+0000 7f8c6dcec640 -1 rbd::mirror::NamespaceReplayer: 0x55d546e5a4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.603 DEBUG:teuthology.parallel:result is None 2023-11-24T09:11:10.603 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-11-24T09:11:10.613 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-11-24T09:11:10.613 DEBUG:teuthology.orchestra.run.smithi164:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-11-24T09:11:10.665 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-11-24T09:11:10.666 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-11-24T09:11:10.729 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-11-24T09:11:10.729 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 20 2023-11-24T09:11:10.729 INFO:tasks.rbd_mirror_thrash:done joining 2023-11-24T09:11:10.730 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-11-24T09:11:10.765 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-11-24T09:11:10.765 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 21 2023-11-24T09:11:10.765 INFO:tasks.rbd_mirror_thrash:done joining 2023-11-24T09:11:10.766 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-24T09:11:10.785 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-11-24T09:11:10.786 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:11:10.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67dcb9d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 76078) UID: 1000 2023-11-24T09:11:10.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67dc39c640 -1 rbd::mirror::InstanceWatcher: 0x5560b65a9880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6520 object: (2) No such file or directory 2023-11-24T09:11:10.787 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67dc39c640 -1 rbd::mirror::InstanceWatcher: 0x5560b65a8380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6520 object: (2) No such file or directory 2023-11-24T09:11:10.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67d5b8f640 -1 rbd::mirror::InstanceWatcher: 0x5560b65a9880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:10.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67d5b8f640 -1 rbd::mirror::InstanceWatcher: 0x5560b65a8380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:10.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67ceb81640 -1 rbd::mirror::MirrorStatusUpdater 0x5560b656e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67ceb81640 -1 rbd::mirror::MirrorStatusUpdater 0x5560b656e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67dc39c640 -1 rbd::mirror::NamespaceReplayer: 0x5560b6524820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67dc39c640 -1 rbd::mirror::NamespaceReplayer: 0x5560b65249c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67d538e640 -1 rbd::mirror::MirrorStatusUpdater 0x5560b656e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67d538e640 -1 rbd::mirror::MirrorStatusUpdater 0x5560b656e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:10.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67dbb9b640 -1 rbd::mirror::NamespaceReplayer: 0x5560b6524820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:10.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:10.786+0000 7f67dbb9b640 -1 rbd::mirror::NamespaceReplayer: 0x5560b65249c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:11.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.294+0000 7f67dc39c640 -1 rbd::mirror::LeaderWatcher: 0x5560b6956300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:11.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.294+0000 7f67c7b73640 -1 rbd::mirror::InstanceWatcher: 0x5560b65a9180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:11.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.294+0000 7f67c1366640 -1 rbd::mirror::MirrorStatusUpdater 0x5560b6770a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:11.298 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.294+0000 7f67dc39c640 -1 rbd::mirror::NamespaceReplayer: 0x5560b69584e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:11.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.298+0000 7f67dbb9b640 -1 rbd::mirror::LeaderWatcher: 0x5560b62fdb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:11.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.298+0000 7f67dbb9b640 -1 rbd::mirror::InstanceWatcher: 0x5560b62ad880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6520 object: (2) No such file or directory 2023-11-24T09:11:11.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.302+0000 7f67d5b8f640 -1 rbd::mirror::InstanceWatcher: 0x5560b62ad880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:11.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.302+0000 7f67cf382640 -1 rbd::mirror::MirrorStatusUpdater 0x5560b6f806c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:11.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.302+0000 7f67dbb9b640 -1 rbd::mirror::NamespaceReplayer: 0x5560b6524000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:11.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.302+0000 7f67d5b8f640 -1 rbd::mirror::MirrorStatusUpdater 0x5560b6f32b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:11.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi164.stderr:2023-11-24T09:11:11.302+0000 7f67dc39c640 -1 rbd::mirror::NamespaceReplayer: 0x5560b6524000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:11.619 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:11.618+0000 7fd249105640 -1 rbd::mirror::PoolWatcher: 0x55a20837a3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:11.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:11.618+0000 7fd2428f8640 -1 rbd::mirror::PoolWatcher: 0x55a20837b860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:11.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:11.618+0000 7fd25d12d640 -1 rbd::mirror::LeaderWatcher: 0x55a20991c300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-11-24T09:11:11.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.894+0000 7f87bfb14640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558610e9a6c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-24T09:11:11.896 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.894+0000 7f87c0315640 -1 rbd::mirror::PoolWatcher: 0x55860fa21a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:11.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.894+0000 7f87b9b08640 -1 rbd::mirror::PoolWatcher: 0x558610f3c000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-24T09:11:11.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.894+0000 7f87c6321640 -1 rbd::mirror::LeaderWatcher: 0x55861163bb00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-11-24T09:11:11.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.894+0000 7f87c6321640 -1 rbd::mirror::LeaderWatcher: 0x55861163bb00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-11-24T09:11:11.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.894+0000 7f87c6b22640 -1 rbd::mirror::LeaderWatcher: 0x55861163bb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:11.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.894+0000 7f87c6b22640 -1 rbd::mirror::InstanceWatcher: 0x55861091f880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6460 object: (2) No such file or directory 2023-11-24T09:11:11.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.894+0000 7f87bfb14640 -1 rbd::mirror::InstanceWatcher: 0x55861091f880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:11.898 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.898+0000 7f87b9307640 -1 rbd::mirror::MirrorStatusUpdater 0x5586116266c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:11.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.898+0000 7f87c6b22640 -1 rbd::mirror::NamespaceReplayer: 0x55861099c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:11.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.898+0000 7f87bfb14640 -1 rbd::mirror::MirrorStatusUpdater 0x558611584b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:11.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.898+0000 7f87c6321640 -1 rbd::mirror::NamespaceReplayer: 0x55861099c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:11.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.898+0000 7f87ca367880 -1 rbd::mirror::Mirror: 0x55860fcebd40 update_pool_replayers: restarting failed pool replayer for uuid: dc1c9e8c-3321-4ffa-8219-1fcd6faafc9e cluster: cluster1 client: client.mirror 2023-11-24T09:11:11.901 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.898+0000 7f87c6b22640 -1 rbd::mirror::LeaderWatcher: 0x558610db6300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:11.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.898+0000 7f87c6b22640 -1 rbd::mirror::InstanceWatcher: 0x558610e85180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6462 object: (2) No such file or directory 2023-11-24T09:11:11.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.898+0000 7f87b2afa640 -1 rbd::mirror::InstanceWatcher: 0x558610e85180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:11.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.902+0000 7f87ac2ed640 -1 rbd::mirror::MirrorStatusUpdater 0x558610c38a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:11.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.902+0000 7f87c6b22640 -1 rbd::mirror::NamespaceReplayer: 0x558610dba4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:11.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.902+0000 7f87b22f9640 -1 rbd::mirror::MirrorStatusUpdater 0x558610c006c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:11.903 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.902+0000 7f87c6321640 -1 rbd::mirror::NamespaceReplayer: 0x558610dba4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:11.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:11.934+0000 7f87ca367880 -1 rbd::mirror::PoolReplayer: 0x5586109bc000 init: error accessing local pool 4: (2) No such file or directory 2023-11-24T09:11:12.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:12.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:12.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:12.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:12.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.842+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.842+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.842+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.842+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.976 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~197s 2023-11-24T09:11:13.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:13.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:13.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:13.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:13.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.846+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.846+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:14.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:14.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:14.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:14.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.157 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.154+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.214 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.298 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.330 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.350 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.352 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.370+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.405 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.666+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.771 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.806+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.842+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.842+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.848 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.846+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.846+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.846+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.872 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.893 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.895 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.896 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.897 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.898 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.899 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.900 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.901 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.902 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.903 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.922 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.989 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.990 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.991 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.994 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.995 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:15.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:15.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:15.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:15.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.029 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.031 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.050+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.054+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.058+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.064 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.062+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.066+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.070+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.076 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.077 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.074+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.078 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.079 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.078+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.081 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.082 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.083 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.084 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.085 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.082+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.086 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.087 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.089 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.086+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.090+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.094+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.101 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.098+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.102+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.106+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.110+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.113 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.114 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.115 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.114+0000 7fd440af0640 -1 rbd::mirror::RemotePoolPoller: 0x55a2b6571860 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-11-24T09:11:16.116 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.114+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.114+0000 7fd4402ef640 -1 rbd::mirror::RemotePoolPoller: 0x55a2b6571860 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-11-24T09:11:16.117 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.114+0000 7fd4402ef640 -1 rbd::mirror::PoolReplayer: 0x55a2b7432000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-11-24T09:11:16.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.117 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd44db0a640 -1 rbd::mirror::InstanceWatcher: 0x55a2b74a3500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6587 object: (2) No such file or directory 2023-11-24T09:11:16.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd44db0a640 -1 rbd::mirror::InstanceWatcher: 0x55a2b74a2a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6587 object: (2) No such file or directory 2023-11-24T09:11:16.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd4472fd640 -1 rbd::mirror::InstanceWatcher: 0x55a2b74a3500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:16.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd446afc640 -1 rbd::mirror::InstanceWatcher: 0x55a2b74a2a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:16.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.119 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd4402ef640 -1 rbd::mirror::MirrorStatusUpdater 0x55a2b74c2a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd4402ef640 -1 rbd::mirror::MirrorStatusUpdater 0x55a2b74c25a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd44d309640 -1 rbd::mirror::NamespaceReplayer: 0x55a2b8320820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd44d309640 -1 rbd::mirror::NamespaceReplayer: 0x55a2b83209c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.120 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.121 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd4472fd640 -1 rbd::mirror::MirrorStatusUpdater 0x55a2b74c2240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd4472fd640 -1 rbd::mirror::MirrorStatusUpdater 0x55a2b74c2360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd44db0a640 -1 rbd::mirror::NamespaceReplayer: 0x55a2b8320820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.118+0000 7fd44db0a640 -1 rbd::mirror::NamespaceReplayer: 0x55a2b83209c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.122 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.123 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.124 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.122+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.126+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.130+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.134+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.138+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.142 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.144 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.145 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.142+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.146 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.147 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.148 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.149 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.146+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.150 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.151 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.152 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.153 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.150+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.154 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.156 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.158+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.162+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.162+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.166+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.166+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.170 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.170+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.173 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.170+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.174+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.174+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.180 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.178+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.178+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.182+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.182+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.186+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.189 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.186+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.192 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.190+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.193 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.190+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.194+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.194+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.197 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.198+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.198+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.202 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.202+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.205 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.202+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.207 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.208 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.206+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.206+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.210+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.210+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.216 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.214+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.214+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.218+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.221 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.218+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.222+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.222+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.226+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.229 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.226+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.226+0000 7fd432ad4640 -1 rbd::mirror::RemotePoolPoller: 0x55a2b7822000 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-11-24T09:11:16.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.226+0000 7fd4332d5640 -1 rbd::mirror::RemotePoolPoller: 0x55a2b7822000 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-11-24T09:11:16.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.226+0000 7fd4332d5640 -1 rbd::mirror::PoolReplayer: 0x55a2b8274600 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-11-24T09:11:16.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.231 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.230+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.233 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.230+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.234+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.236 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.237 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.234+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.240 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.238+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.238+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.244 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.242+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.242+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.246+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.246+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.250+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.250+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd44db0a640 -1 rbd::mirror::LeaderWatcher: 0x55a2b8274300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:16.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.256 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd44db0a640 -1 rbd::mirror::InstanceWatcher: 0x55a2b742f880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6587 object: (2) No such file or directory 2023-11-24T09:11:16.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd446afc640 -1 rbd::mirror::InstanceWatcher: 0x55a2b742f880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:16.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.254+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd4402ef640 -1 rbd::mirror::MirrorStatusUpdater 0x55a2b828a480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd44db0a640 -1 rbd::mirror::NamespaceReplayer: 0x55a2b8320000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd4472fd640 -1 rbd::mirror::MirrorStatusUpdater 0x55a2b80e6c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.254+0000 7fd44d309640 -1 rbd::mirror::NamespaceReplayer: 0x55a2b8320000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.258+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.258+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.262+0000 7fd44d309640 -1 rbd::mirror::InstanceWatcher: 0x55a2b77daa80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6590 object: (2) No such file or directory 2023-11-24T09:11:16.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.262+0000 7fd439ae2640 -1 rbd::mirror::InstanceWatcher: 0x55a2b77daa80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:16.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.264 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.262+0000 7fd4392e1640 -1 rbd::mirror::MirrorStatusUpdater 0x55a2b76d66c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.262+0000 7fd44d309640 -1 rbd::mirror::NamespaceReplayer: 0x55a2b7812680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.262+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.262+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.266+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.266+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.270 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.270+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.270+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.274 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.276 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.274+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.274+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.280 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.281 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.278+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.278+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.283 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.284 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.282+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.282+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.286+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.289 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.286+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.291 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.290+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.290+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.294 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.295 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.296 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.294+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.297 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.294+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.299 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.300 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.298+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.301 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.298+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.303 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.304 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.302+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.302+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.306 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.306+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.306+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.313 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.310+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.310+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.314+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.314+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.318+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.321 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.318+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.324 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.322+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.326 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.322+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.327 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.328 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.326+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.326+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.330+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.330+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.337 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.334+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.334+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.338+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.338+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.342+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.342+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.346+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.346+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.350+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.350+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.354+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.354+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.360 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.358+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.358+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.362+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.362+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.367 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.366+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.366+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.370+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.374+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.374+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.378+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.378+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.382 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.384 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.382+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.382+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.386+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.386+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.390+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.393 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.390+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.394 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.394+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.394+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.398+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.401 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.398+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.403 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.402+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.402+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.408 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.406+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.409 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.406+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.411 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.412 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.410+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.410+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.414+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.417 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.414+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.418+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.418+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.423 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.424 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.422+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.422+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.426+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.426+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.430+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.430+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.434+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.434+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.438+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.438+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.442+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.442+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.446+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.446+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.450+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.450+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.454+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.454+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.458+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.458+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.462+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.462+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.469 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.466+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.466+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.470+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.470+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.474+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.474+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.478+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.478+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.482+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.485 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.482+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.486+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.486+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.490+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.492 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.490+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.494+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.494+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.498+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.498+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.502 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.502+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.505 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.502+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.506 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.508 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.506+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.506+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.512 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.510+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.513 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.510+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.514+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.514+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.518+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.518+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.522+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.522+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.528 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.526+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.526+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.530+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.530+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.534+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.534+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.538+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.538+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.542+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.542+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.546 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.546+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.549 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.546+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.550+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.550+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.554 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.555 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.554+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.554+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.558+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.558+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.562+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.562+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.566+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.570 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.566+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.570+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.570+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.574+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.574+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.579 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.578+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.578+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.582 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.582+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.582+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.586 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.586+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.586+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.590+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.590+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.594+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.594+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.600 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.598+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.601 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.598+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.603 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.602+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.602+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.607 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.608 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.606+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.606+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.610+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.613 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.610+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.614+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.614+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.618+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.618+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.623 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.624 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.622+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.625 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.622+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.626 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.626+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.629 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.626+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.630 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.631 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.632 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.630+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.633 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.630+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.634+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.634+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.641 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.638+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.638+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.642 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.644 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.642+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.642+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.646+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.646+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.650+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.650+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.654 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.655 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.654+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.654+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.659 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.660 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.658+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.661 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.658+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.662 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.663 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.664 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.662+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.665 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.662+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.666 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.667 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.668 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.666+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.670 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.671 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.670+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.670+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.674+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.674+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.678+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.678+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.684 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.685 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.682+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.682+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.686 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.687 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.688 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.686+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.689 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.686+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.690 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.691 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.692 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.690+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.693 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.690+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.695 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.694+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.694+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.698+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.700 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.698+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.702+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.702+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.706+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.709 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.706+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.710+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.713 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.710+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.714+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.714+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.719 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.720 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.718+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.718+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.722+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.722+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.726+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.726+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.732 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.730+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.730+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.734 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.736 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.734+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.734+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.739 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.738+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.738+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.741 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.742 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.744 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.742+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.745 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.742+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.746+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.749 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.746+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.750 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.752 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.750+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.750+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.756 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.754+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.757 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.754+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.758+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.758+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.764 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.762+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.762+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.768 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.766+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.766+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.770 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.773 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.770+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.770+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.774+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.774+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.778 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.781 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.778+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.782 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.778+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.782+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.782+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.786+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.789 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.786+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.790+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.793 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.790+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.794 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.796 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.794+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.794+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.800 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.798+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.798+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.802+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.802+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.808 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.810 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.806+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.811 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.813 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.810+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.810+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.815 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.816 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.814+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.814+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.820 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.818+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.818+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.822+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.822+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.828 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.826+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.826+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.830 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.831 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.833 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.830+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.830+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.834 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.836 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.834+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.834+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.838+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.838+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.842+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.842+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.842+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.842+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.850+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.850+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.854+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.857 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.854+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.858 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.859 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.860 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.858+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.861 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.858+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.862 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.863 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.864 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.862+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.862+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.866 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.866+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.869 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.866+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.870 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.873 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.874 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.870+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.875 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.870+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.876 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.877 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.878 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.874+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.879 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.874+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.880 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.881 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.882 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.883 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.878+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.878+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.885 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.882+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.882+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.888 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.890 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.891 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2023-11-24T09:11:16.891 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-24T09:11:16.904 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-11-24T09:11:16.904 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:11:16.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.904 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.886+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.886+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.905 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.890+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.890+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.906 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.907 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.894+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.894+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.908 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.898+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.898+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.902+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.912 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi164.stderr:2023-11-24T09:11:16.902+0000 7fd44e30b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 76987) UID: 1000 2023-11-24T09:11:16.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.902+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.906+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.906+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.915 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.916 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.910+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.910+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.917 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.918 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.914+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.914+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.919 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.920 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.918+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.921 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.918+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.923 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.922+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.922+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.926+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2023-11-24T09:11:16.929 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-24T09:11:16.941 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2023-11-24T09:11:16.941 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:11:16.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.926+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.930+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.930+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.944 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.934+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.934+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.938+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.948 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.938+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.948 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.938+0000 7fd25e12f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 73431) UID: 1000 2023-11-24T09:11:16.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.949 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd25d92e640 -1 rbd::mirror::InstanceWatcher: 0x55a209527500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6335 object: (2) No such file or directory 2023-11-24T09:11:16.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd25d92e640 -1 rbd::mirror::InstanceWatcher: 0x55a209526e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6332 object: (2) No such file or directory 2023-11-24T09:11:16.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd25d12d640 -1 rbd::mirror::InstanceWatcher: 0x55a209516000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6332 object: (2) No such file or directory 2023-11-24T09:11:16.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd249906640 -1 rbd::mirror::InstanceWatcher: 0x55a209527500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:16.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd256920640 -1 rbd::mirror::InstanceWatcher: 0x55a209526e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:16.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd256920640 -1 rbd::mirror::InstanceWatcher: 0x55a209516000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:16.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.951 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.942+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd2430f9640 -1 rbd::mirror::MirrorStatusUpdater 0x55a209568480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.942+0000 7fd25d92e640 -1 rbd::mirror::NamespaceReplayer: 0x55a2099204e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd250113640 -1 rbd::mirror::MirrorStatusUpdater 0x55a209568360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd250113640 -1 rbd::mirror::MirrorStatusUpdater 0x55a2097927e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd25d12d640 -1 rbd::mirror::NamespaceReplayer: 0x55a20950c820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd25d12d640 -1 rbd::mirror::NamespaceReplayer: 0x55a20950c9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd249105640 -1 rbd::mirror::MirrorStatusUpdater 0x55a2097486c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd25d12d640 -1 rbd::mirror::NamespaceReplayer: 0x55a2099204e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.953 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd257121640 -1 rbd::mirror::MirrorStatusUpdater 0x55a209568000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd257121640 -1 rbd::mirror::MirrorStatusUpdater 0x55a209568120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd25d12d640 -1 rbd::mirror::NamespaceReplayer: 0x55a20950c820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd25d12d640 -1 rbd::mirror::NamespaceReplayer: 0x55a20950c9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.946+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.955 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.946+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.956 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.950+0000 7fd25d12d640 -1 rbd::mirror::LeaderWatcher: 0x55a2092e1b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:16.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.950+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.958 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.950+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.950+0000 7fd25d12d640 -1 rbd::mirror::InstanceWatcher: 0x55a20928d880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6332 object: (2) No such file or directory 2023-11-24T09:11:16.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.954+0000 7fd256920640 -1 rbd::mirror::InstanceWatcher: 0x55a20928d880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:16.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.954+0000 7fd250914640 -1 rbd::mirror::MirrorStatusUpdater 0x55a209f586c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.954+0000 7fd25d12d640 -1 rbd::mirror::NamespaceReplayer: 0x55a20950c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.954+0000 7fd257121640 -1 rbd::mirror::MirrorStatusUpdater 0x55a209f04b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:16.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi164.stderr:2023-11-24T09:11:16.954+0000 7fd25d92e640 -1 rbd::mirror::NamespaceReplayer: 0x55a20950c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:16.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.954+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.954+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.962 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.958+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.958+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.962+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.962+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.968 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.966+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.969 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.966+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.971 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.970+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.970+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.974+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.974+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.980 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.981 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.978+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.978+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.985 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2023-11-24T09:11:16.986 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-24T09:11:16.997 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2023-11-24T09:11:16.997 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:11:16.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.982+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.982+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:16.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:16.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.986+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.986+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.004 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.990+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.990+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.994+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.994+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.009 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.998+0000 7f87c7323640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 75215) UID: 1000 2023-11-24T09:11:17.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:16.998+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:16.998+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.002+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.002+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.015 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.016 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.006+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.006+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.010+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.010+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.021 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.014+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.014+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2023-11-24T09:11:17.026 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-24T09:11:17.037 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-11-24T09:11:17.038 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:11:17.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.018+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.018+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.022+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.022+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.043 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.026+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.026+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.030+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.048 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.030+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.051 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.034+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.034+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bfad8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 75423) UID: 1000 2023-11-24T09:11:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614b36a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bf2d7640 -1 rbd::mirror::InstanceWatcher: 0x55614af68000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6197 object: (2) No such file or directory 2023-11-24T09:11:17.053 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bead6640 -1 rbd::mirror::InstanceWatcher: 0x55614af68700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6197 object: (2) No such file or directory 2023-11-24T09:11:17.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.054 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0b82c9640 -1 rbd::mirror::InstanceWatcher: 0x55614af68000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0b82c9640 -1 rbd::mirror::InstanceWatcher: 0x55614af68700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bf2d7640 -1 rbd::mirror::InstanceWatcher: 0x55614af64e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6200 object: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0b22bd640 -1 rbd::mirror::MirrorStatusUpdater 0x55614afb0360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0b22bd640 -1 rbd::mirror::MirrorStatusUpdater 0x55614afb06c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bead6640 -1 rbd::mirror::NamespaceReplayer: 0x55614af56820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bead6640 -1 rbd::mirror::NamespaceReplayer: 0x55614af569c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bead6640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bead6640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.038+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.056 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0b82c9640 -1 rbd::mirror::MirrorStatusUpdater 0x55614afb0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0b82c9640 -1 rbd::mirror::MirrorStatusUpdater 0x55614afb0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bf2d7640 -1 rbd::mirror::NamespaceReplayer: 0x55614af56820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0bf2d7640 -1 rbd::mirror::NamespaceReplayer: 0x55614af569c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.038+0000 7fd0aaaae640 -1 rbd::mirror::InstanceWatcher: 0x55614af64e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0a4aa2640 -1 rbd::mirror::MirrorStatusUpdater 0x55614afb0480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0bf2d7640 -1 rbd::mirror::NamespaceReplayer: 0x55614b36c4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.057 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0aaaae640 -1 rbd::mirror::MirrorStatusUpdater 0x55614b17e7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0bf2d7640 -1 rbd::mirror::NamespaceReplayer: 0x55614b36c4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.042+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.042+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.046+0000 7fd0bf2d7640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.060 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.046+0000 7fd0bead6640 -1 rbd::mirror::LeaderWatcher: 0x55614ad33b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:17.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.061 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.046+0000 7fd0bead6640 -1 rbd::mirror::InstanceWatcher: 0x55614ac0d880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6197 object: (2) No such file or directory 2023-11-24T09:11:17.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.046+0000 7fd0b8aca640 -1 rbd::mirror::InstanceWatcher: 0x55614ac0d880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:17.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.046+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.062 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.046+0000 7fd0b1abc640 -1 rbd::mirror::MirrorStatusUpdater 0x55614b9b86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.046+0000 7fd0bead6640 -1 rbd::mirror::NamespaceReplayer: 0x55614af56000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.046+0000 7fd0b82c9640 -1 rbd::mirror::MirrorStatusUpdater 0x55614b93cb40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.063 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi164.stderr:2023-11-24T09:11:17.046+0000 7fd0bf2d7640 -1 rbd::mirror::NamespaceReplayer: 0x55614af56000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.065 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.050+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.054+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.066 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.058+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.062+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.068 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.069 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.066+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.070 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.071 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.073 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.070+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.074 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2023-11-24T09:11:17.074 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-24T09:11:17.087 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2023-11-24T09:11:17.087 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:11:17.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.088 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.090 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.074+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.078+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.091 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.082+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.092 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.093 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi164.stderr:2023-11-24T09:11:17.086+0000 7f8c6e4ed640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 67310) UID: 1000 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.086+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.094 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.090+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.095 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.096 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.097 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.094+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.099 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.100 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.102 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.098+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.103 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.104 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.102+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.107 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.108 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.109 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.106+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.110 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.111 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2023-11-24T09:11:17.111 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-24T09:11:17.123 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-11-24T09:11:17.123 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:11:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.110+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.125 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.126 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.114+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.118+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.127 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.128 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi164.stderr:2023-11-24T09:11:17.122+0000 7f4facdda640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 76311) UID: 1000 2023-11-24T09:11:17.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.122+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.126+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.130 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.131 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.132 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.130+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.134 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.134+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.139 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.141 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.138+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.143 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2023-11-24T09:11:17.143 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-24T09:11:17.155 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-11-24T09:11:17.155 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:11:17.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.155 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.158 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.142+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.146+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.159 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.150+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66e015d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 76850) UID: 1000 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df95c640 -1 rbd::mirror::InstanceWatcher: 0x555acac12000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6284 object: (2) No such file or directory 2023-11-24T09:11:17.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df15b640 -1 rbd::mirror::InstanceWatcher: 0x555acaba4a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6284 object: (2) No such file or directory 2023-11-24T09:11:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66cb934640 -1 rbd::mirror::InstanceWatcher: 0x555acac3d500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66d894e640 -1 rbd::mirror::InstanceWatcher: 0x555acac12000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.154+0000 7f66d914f640 -1 rbd::mirror::InstanceWatcher: 0x555acaba4a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66d2942640 -1 rbd::mirror::MirrorStatusUpdater 0x555acac06360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66d2942640 -1 rbd::mirror::MirrorStatusUpdater 0x555acac06480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df15b640 -1 rbd::mirror::NamespaceReplayer: 0x555acba20820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df15b640 -1 rbd::mirror::NamespaceReplayer: 0x555acba209c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66d894e640 -1 rbd::mirror::MirrorStatusUpdater 0x555acac06000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66d894e640 -1 rbd::mirror::MirrorStatusUpdater 0x555acac06120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df95c640 -1 rbd::mirror::NamespaceReplayer: 0x555acba20820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df95c640 -1 rbd::mirror::NamespaceReplayer: 0x555acba209c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df15b640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df15b640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df15b640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df15b640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_acquire_leader_lock: error acquiring lock: (2) No such file or directory 2023-11-24T09:11:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df15b640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_post_acquire_leader_lock: error acquiring leader lock: (2) No such file or directory 2023-11-24T09:11:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.158+0000 7f66df95c640 -1 rbd::mirror::LeaderWatcher: 0x555acb98a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-24T09:11:17.163 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.162+0000 7f66df95c640 -1 rbd::mirror::InstanceWatcher: 0x555acab21880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6284 object: (2) No such file or directory 2023-11-24T09:11:17.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.162+0000 7f66d914f640 -1 rbd::mirror::InstanceWatcher: 0x555acab21880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-24T09:11:17.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.162+0000 7f66d2942640 -1 rbd::mirror::MirrorStatusUpdater 0x555acb8346c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.162+0000 7f66df95c640 -1 rbd::mirror::NamespaceReplayer: 0x555acba20000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.162+0000 7f66d914f640 -1 rbd::mirror::MirrorStatusUpdater 0x555acb794c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-24T09:11:17.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi164.stderr:2023-11-24T09:11:17.162+0000 7f66df15b640 -1 rbd::mirror::NamespaceReplayer: 0x555acba20000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-24T09:11:17.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2023-11-24T09:11:17.183 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-24T09:11:17.250 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-11-24T09:11:17.250 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:17.555 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:17.555 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:17.573 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1753,"stamp":"2023-11-24T09:11:16.927668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":239,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006468,"kb_used_data":419496,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109052,"statfs":{"total":289910292480,"available":286831669248,"internally_reserved":0,"allocated":429563904,"data_stored":420638979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":321,"apply_latency_ms":321,"commit_latency_ns":321000000,"apply_latency_ns":321000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2376,"num_read_kb":-2246,"num_write":-465,"num_write_kb":-350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.932685"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:55.317644+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:10:55.317644+0000","last_peered":"2023-11-24T09:10:55.317644+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:10:55.317644+0000","last_undegraded":"2023-11-24T09:10:55.317644+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:50.315352+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:10:50.315352+0000","last_peered":"2023-11-24T09:10:50.315352+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:10:50.315352+0000","last_undegraded":"2023-11-24T09:10:50.315352+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:33.789832+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:10:33.789832+0000","last_peered":"2023-11-24T09:10:33.789832+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:10:33.789832+0000","last_undegraded":"2023-11-24T09:10:33.789832+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:33.782707+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:10:33.782707+0000","last_peered":"2023-11-24T09:10:33.782707+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:10:33.782707+0000","last_undegraded":"2023-11-24T09:10:33.782707+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:33.785726+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:10:33.785726+0000","last_peered":"2023-11-24T09:10:33.785726+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:10:33.785726+0000","last_undegraded":"2023-11-24T09:10:33.785726+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":517,"reported_epoch":502,"state":"active+clean","last_fresh":"2023-11-24T09:10:33.796145+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:10:33.796145+0000","last_peered":"2023-11-24T09:10:33.796145+0000","last_clean":"2023-11-24T09:10:33.796145+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:10:33.796145+0000","last_undegraded":"2023-11-24T09:10:33.796145+0000","last_fullsized":"2023-11-24T09:10:33.796145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":518,"reported_epoch":502,"state":"active+clean","last_fresh":"2023-11-24T09:10:33.799602+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:10:33.799602+0000","last_peered":"2023-11-24T09:10:33.799602+0000","last_clean":"2023-11-24T09:10:33.799602+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:10:33.799602+0000","last_undegraded":"2023-11-24T09:10:33.799602+0000","last_fullsized":"2023-11-24T09:10:33.799602+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":518,"reported_epoch":502,"state":"active+clean","last_fresh":"2023-11-24T09:10:33.783626+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:10:33.783626+0000","last_peered":"2023-11-24T09:10:33.783626+0000","last_clean":"2023-11-24T09:10:33.783626+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:10:33.783626+0000","last_undegraded":"2023-11-24T09:10:33.783626+0000","last_fullsized":"2023-11-24T09:10:33.783626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":653,"reported_epoch":502,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:10:33.788414+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:10:33.788414+0000","last_peered":"2023-11-24T09:10:33.788414+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:10:33.788414+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771416,"kb_used_data":5840,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600424,"statfs":{"total":96636764160,"available":95846834176,"internally_reserved":0,"allocated":5980160,"data_stored":2982905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":143,"apply_latency_ms":143,"commit_latency_ns":143000000,"apply_latency_ns":143000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.48499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738872,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840284,"kb_used_data":6420,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531556,"statfs":{"total":96636764160,"available":95776313344,"internally_reserved":0,"allocated":6574080,"data_stored":3573273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":177,"apply_latency_ms":177,"commit_latency_ns":177000000,"apply_latency_ns":177000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:17.574 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:17.855 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:17.855 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:17.874 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1753,"stamp":"2023-11-24T09:11:16.927668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":239,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006468,"kb_used_data":419496,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109052,"statfs":{"total":289910292480,"available":286831669248,"internally_reserved":0,"allocated":429563904,"data_stored":420638979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":321,"apply_latency_ms":321,"commit_latency_ns":321000000,"apply_latency_ns":321000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2376,"num_read_kb":-2246,"num_write":-465,"num_write_kb":-350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.932685"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:55.317644+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:10:55.317644+0000","last_peered":"2023-11-24T09:10:55.317644+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:10:55.317644+0000","last_undegraded":"2023-11-24T09:10:55.317644+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:50.315352+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:10:50.315352+0000","last_peered":"2023-11-24T09:10:50.315352+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:10:50.315352+0000","last_undegraded":"2023-11-24T09:10:50.315352+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:33.789832+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:10:33.789832+0000","last_peered":"2023-11-24T09:10:33.789832+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:10:33.789832+0000","last_undegraded":"2023-11-24T09:10:33.789832+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:33.782707+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:10:33.782707+0000","last_peered":"2023-11-24T09:10:33.782707+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:10:33.782707+0000","last_undegraded":"2023-11-24T09:10:33.782707+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:33.785726+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:10:33.785726+0000","last_peered":"2023-11-24T09:10:33.785726+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:10:33.785726+0000","last_undegraded":"2023-11-24T09:10:33.785726+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":517,"reported_epoch":502,"state":"active+clean","last_fresh":"2023-11-24T09:10:33.796145+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:10:33.796145+0000","last_peered":"2023-11-24T09:10:33.796145+0000","last_clean":"2023-11-24T09:10:33.796145+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:10:33.796145+0000","last_undegraded":"2023-11-24T09:10:33.796145+0000","last_fullsized":"2023-11-24T09:10:33.796145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":518,"reported_epoch":502,"state":"active+clean","last_fresh":"2023-11-24T09:10:33.799602+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:10:33.799602+0000","last_peered":"2023-11-24T09:10:33.799602+0000","last_clean":"2023-11-24T09:10:33.799602+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:10:33.799602+0000","last_undegraded":"2023-11-24T09:10:33.799602+0000","last_fullsized":"2023-11-24T09:10:33.799602+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":518,"reported_epoch":502,"state":"active+clean","last_fresh":"2023-11-24T09:10:33.783626+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:10:33.783626+0000","last_peered":"2023-11-24T09:10:33.783626+0000","last_clean":"2023-11-24T09:10:33.783626+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:10:33.783626+0000","last_undegraded":"2023-11-24T09:10:33.783626+0000","last_fullsized":"2023-11-24T09:10:33.783626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":653,"reported_epoch":502,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:10:33.788414+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:10:33.788414+0000","last_peered":"2023-11-24T09:10:33.788414+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:10:33.788414+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771416,"kb_used_data":5840,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600424,"statfs":{"total":96636764160,"available":95846834176,"internally_reserved":0,"allocated":5980160,"data_stored":2982905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":143,"apply_latency_ms":143,"commit_latency_ns":143000000,"apply_latency_ns":143000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.48499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738872,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840284,"kb_used_data":6420,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531556,"statfs":{"total":96636764160,"available":95776313344,"internally_reserved":0,"allocated":6574080,"data_stored":3573273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":177,"apply_latency_ms":177,"commit_latency_ns":177000000,"apply_latency_ns":177000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:17.875 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:18.371 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:18.371 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:18.389 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1753,"stamp":"2023-11-24T09:11:16.927668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":239,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006468,"kb_used_data":419496,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109052,"statfs":{"total":289910292480,"available":286831669248,"internally_reserved":0,"allocated":429563904,"data_stored":420638979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":321,"apply_latency_ms":321,"commit_latency_ns":321000000,"apply_latency_ns":321000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2376,"num_read_kb":-2246,"num_write":-465,"num_write_kb":-350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.932685"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:55.317644+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:10:55.317644+0000","last_peered":"2023-11-24T09:10:55.317644+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:10:55.317644+0000","last_undegraded":"2023-11-24T09:10:55.317644+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:50.315352+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:10:50.315352+0000","last_peered":"2023-11-24T09:10:50.315352+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:10:50.315352+0000","last_undegraded":"2023-11-24T09:10:50.315352+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:33.789832+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:10:33.789832+0000","last_peered":"2023-11-24T09:10:33.789832+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:10:33.789832+0000","last_undegraded":"2023-11-24T09:10:33.789832+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":522,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:33.782707+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:10:33.782707+0000","last_peered":"2023-11-24T09:10:33.782707+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:10:33.782707+0000","last_undegraded":"2023-11-24T09:10:33.782707+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":502,"state":"active+undersized","last_fresh":"2023-11-24T09:10:33.785726+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:10:33.785726+0000","last_peered":"2023-11-24T09:10:33.785726+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:10:33.785726+0000","last_undegraded":"2023-11-24T09:10:33.785726+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":517,"reported_epoch":502,"state":"active+clean","last_fresh":"2023-11-24T09:10:33.796145+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:10:33.796145+0000","last_peered":"2023-11-24T09:10:33.796145+0000","last_clean":"2023-11-24T09:10:33.796145+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:10:33.796145+0000","last_undegraded":"2023-11-24T09:10:33.796145+0000","last_fullsized":"2023-11-24T09:10:33.796145+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":518,"reported_epoch":502,"state":"active+clean","last_fresh":"2023-11-24T09:10:33.799602+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:10:33.799602+0000","last_peered":"2023-11-24T09:10:33.799602+0000","last_clean":"2023-11-24T09:10:33.799602+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:10:33.799602+0000","last_undegraded":"2023-11-24T09:10:33.799602+0000","last_fullsized":"2023-11-24T09:10:33.799602+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":518,"reported_epoch":502,"state":"active+clean","last_fresh":"2023-11-24T09:10:33.783626+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:10:33.783626+0000","last_peered":"2023-11-24T09:10:33.783626+0000","last_clean":"2023-11-24T09:10:33.783626+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:10:33.783626+0000","last_undegraded":"2023-11-24T09:10:33.783626+0000","last_fullsized":"2023-11-24T09:10:33.783626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":653,"reported_epoch":502,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:10:33.788414+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:10:33.788414+0000","last_peered":"2023-11-24T09:10:33.788414+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:10:33.788414+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771416,"kb_used_data":5840,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600424,"statfs":{"total":96636764160,"available":95846834176,"internally_reserved":0,"allocated":5980160,"data_stored":2982905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":143,"apply_latency_ms":143,"commit_latency_ns":143000000,"apply_latency_ns":143000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.48499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738872,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840284,"kb_used_data":6420,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531556,"statfs":{"total":96636764160,"available":95776313344,"internally_reserved":0,"allocated":6574080,"data_stored":3573273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":177,"apply_latency_ms":177,"commit_latency_ns":177000000,"apply_latency_ns":177000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:18.391 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:11:18.391 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 522, 'reported_epoch': 502, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:10:55.317644+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:10:55.317644+0000', 'last_peered': '2023-11-24T09:10:55.317644+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:10:55.317644+0000', 'last_undegraded': '2023-11-24T09:10:55.317644+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:18.391 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:11:18.391 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 522, 'reported_epoch': 502, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:10:50.315352+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:10:50.315352+0000', 'last_peered': '2023-11-24T09:10:50.315352+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:10:50.315352+0000', 'last_undegraded': '2023-11-24T09:10:50.315352+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:18.391 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:11:18.391 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 522, 'reported_epoch': 502, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:10:33.789832+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:10:33.789832+0000', 'last_peered': '2023-11-24T09:10:33.789832+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:10:33.789832+0000', 'last_undegraded': '2023-11-24T09:10:33.789832+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:18.391 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:11:18.392 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 522, 'reported_epoch': 502, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:10:33.782707+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:10:33.782707+0000', 'last_peered': '2023-11-24T09:10:33.782707+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:10:33.782707+0000', 'last_undegraded': '2023-11-24T09:10:33.782707+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:18.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:11:18.392 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 22, 'reported_epoch': 502, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:10:33.785726+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:10:33.785726+0000', 'last_peered': '2023-11-24T09:10:33.785726+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:10:33.785726+0000', 'last_undegraded': '2023-11-24T09:10:33.785726+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:18.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:11:18.392 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 653, 'reported_epoch': 502, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:10:33.788414+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:10:33.788414+0000', 'last_peered': '2023-11-24T09:10:33.788414+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:10:33.788414+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:18.393 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:11:18.738 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:18.738 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2514,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078623232,"bytes_avail":286831669248,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":50,"modified":"2023-11-24T09:11:16.927715+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"rbd-mirror":{"daemons":{"summary":"","6329":{"start_epoch":40,"start_stamp":"2023-11-24T09:03:56.149251+0000","gid":6329,"addr":"172.21.15.164:0/900483656","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev)","ceph_version_short":"18.0.0-7484-g135a1ac9","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"ubuntu","distro_description":"Ubuntu 22.04.1 LTS","distro_version":"22.04","hostname":"smithi164","id":"mirror.1","instance_id":"6329","kernel_description":"#99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023","kernel_version":"5.15.0-89-generic","mem_swap_kb":"2097148","mem_total_kb":"32746948","os":"Linux"},"task_status":{}},"6457":{"start_epoch":44,"start_stamp":"2023-11-24T09:07:09.074539+0000","gid":6457,"addr":"172.21.15.164:0/3535610523","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 18.0.0-7484-g135a1ac9 (135a1ac939fb4e6d1ef988cf89d0da570e04d3d1) reef (dev)","ceph_version_short":"18.0.0-7484-g135a1ac9","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"ubuntu","distro_description":"Ubuntu 22.04.1 LTS","distro_version":"22.04","hostname":"smithi164","id":"mirror.0","instance_id":"6457","kernel_description":"#99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023","kernel_version":"5.15.0-89-generic","mem_swap_kb":"2097148","mem_total_kb":"32746948","os":"Linux"},"task_status":{}}}}}},"progress_events":{}} 2023-11-24T09:11:18.757 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:11:18.758 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:19.040 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:19.056 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1754,"stamp":"2023-11-24T09:11:18.928358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2376,"num_read_kb":-2246,"num_write":-465,"num_write_kb":-350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003609"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.014550+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:08.014550+0000","last_peered":"2023-11-24T09:11:08.014550+0000","last_clean":"2023-11-24T09:11:08.014550+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:08.014550+0000","last_undegraded":"2023-11-24T09:11:08.014550+0000","last_fullsized":"2023-11-24T09:11:08.014550+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.439},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.54700000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738873,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.873}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:19.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~202s 2023-11-24T09:11:22.057 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:22.360 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:22.378 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1755,"stamp":"2023-11-24T09:11:20.929150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.925237"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.014550+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:08.014550+0000","last_peered":"2023-11-24T09:11:08.014550+0000","last_clean":"2023-11-24T09:11:08.014550+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:08.014550+0000","last_undegraded":"2023-11-24T09:11:08.014550+0000","last_fullsized":"2023-11-24T09:11:08.014550+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.439},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.54700000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738873,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.873}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:22.379 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:22.650 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:22.650 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:22.666 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1755,"stamp":"2023-11-24T09:11:20.929150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.925237"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.014550+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:08.014550+0000","last_peered":"2023-11-24T09:11:08.014550+0000","last_clean":"2023-11-24T09:11:08.014550+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:08.014550+0000","last_undegraded":"2023-11-24T09:11:08.014550+0000","last_fullsized":"2023-11-24T09:11:08.014550+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.439},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.54700000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738873,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.873}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:22.667 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:11:22.667 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:22.668 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:11:22.668 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:22.668 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:11:22.668 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019191+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:08.019191+0000', 'last_peered': '2023-11-24T09:11:08.019191+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:08.019191+0000', 'last_undegraded': '2023-11-24T09:11:08.019191+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:22.668 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:11:22.668 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.009824+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:08.009824+0000', 'last_peered': '2023-11-24T09:11:08.009824+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:08.009824+0000', 'last_undegraded': '2023-11-24T09:11:08.009824+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:22.668 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:11:22.668 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019565+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:08.019565+0000', 'last_peered': '2023-11-24T09:11:08.019565+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:08.019565+0000', 'last_undegraded': '2023-11-24T09:11:08.019565+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:22.668 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:11:22.669 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 654, 'reported_epoch': 503, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:08.015143+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:08.015143+0000', 'last_peered': '2023-11-24T09:11:08.015143+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:08.015143+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:22.669 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:11:22.981 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:22.981 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2518,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-11-24T09:11:18.928467+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-11-24T09:11:22.999 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:11:22.999 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:23.271 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:23.271 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:23.287 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1756,"stamp":"2023-11-24T09:11:22.929864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004126"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.014550+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:08.014550+0000","last_peered":"2023-11-24T09:11:08.014550+0000","last_clean":"2023-11-24T09:11:08.014550+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:08.014550+0000","last_undegraded":"2023-11-24T09:11:08.014550+0000","last_fullsized":"2023-11-24T09:11:08.014550+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.55800000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738873,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.873}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:24.785 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~208s 2023-11-24T09:11:26.289 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:26.592 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:26.610 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1757,"stamp":"2023-11-24T09:11:24.930597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004311"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.55800000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:26.611 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:26.901 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:26.901 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:26.919 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1757,"stamp":"2023-11-24T09:11:24.930597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004311"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.55800000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:26.920 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:11:26.920 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:26.920 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:11:26.920 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:26.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:11:26.921 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019191+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:08.019191+0000', 'last_peered': '2023-11-24T09:11:08.019191+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:08.019191+0000', 'last_undegraded': '2023-11-24T09:11:08.019191+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:26.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:11:26.921 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.009824+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:08.009824+0000', 'last_peered': '2023-11-24T09:11:08.009824+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:08.009824+0000', 'last_undegraded': '2023-11-24T09:11:08.009824+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:26.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:11:26.921 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019565+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:08.019565+0000', 'last_peered': '2023-11-24T09:11:08.019565+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:08.019565+0000', 'last_undegraded': '2023-11-24T09:11:08.019565+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:26.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:11:26.921 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 654, 'reported_epoch': 503, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:08.015143+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:08.015143+0000', 'last_peered': '2023-11-24T09:11:08.015143+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:08.015143+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:26.921 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:11:27.263 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:27.263 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2522,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-11-24T09:11:18.928467+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-11-24T09:11:27.282 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:11:27.282 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:27.564 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:27.565 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:27.584 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1758,"stamp":"2023-11-24T09:11:26.931343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004323"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738875,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.55800000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:30.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~213s 2023-11-24T09:11:30.585 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:30.888 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:30.888 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:30.907 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1759,"stamp":"2023-11-24T09:11:28.932028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004359"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.49299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.46899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:30.908 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:31.195 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:31.196 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:31.211 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1760,"stamp":"2023-11-24T09:11:30.932762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004404"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.49299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.46899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:31.212 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:11:31.213 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:31.213 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:11:31.213 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:31.213 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:11:31.213 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019191+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:08.019191+0000', 'last_peered': '2023-11-24T09:11:08.019191+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:08.019191+0000', 'last_undegraded': '2023-11-24T09:11:08.019191+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:31.213 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:11:31.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.009824+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:08.009824+0000', 'last_peered': '2023-11-24T09:11:08.009824+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:08.009824+0000', 'last_undegraded': '2023-11-24T09:11:08.009824+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:31.214 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:11:31.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019565+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:08.019565+0000', 'last_peered': '2023-11-24T09:11:08.019565+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:08.019565+0000', 'last_undegraded': '2023-11-24T09:11:08.019565+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:31.214 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:11:31.214 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 654, 'reported_epoch': 503, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:08.015143+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:08.015143+0000', 'last_peered': '2023-11-24T09:11:08.015143+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:08.015143+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:31.215 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:11:31.534 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:31.534 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2527,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-11-24T09:11:18.928467+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-11-24T09:11:31.551 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:11:31.552 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:31.812 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:31.812 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:31.829 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1760,"stamp":"2023-11-24T09:11:30.932762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004404"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019191+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:08.019191+0000","last_peered":"2023-11-24T09:11:08.019191+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:08.019191+0000","last_undegraded":"2023-11-24T09:11:08.019191+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.009824+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:08.009824+0000","last_peered":"2023-11-24T09:11:08.009824+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:08.009824+0000","last_undegraded":"2023-11-24T09:11:08.009824+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.49299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.46899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:34.831 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:35.126 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:35.126 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:35.142 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1762,"stamp":"2023-11-24T09:11:34.934218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738877,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.625}]}]},{"osd":1,"up_from":8,"seq":34359738876,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.53600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:35.143 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:35.417 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:35.417 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:35.435 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1762,"stamp":"2023-11-24T09:11:34.934218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738877,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.625}]}]},{"osd":1,"up_from":8,"seq":34359738876,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.53600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:35.436 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:11:35.436 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:35.436 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:11:35.436 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:35.436 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:11:35.436 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:35.436 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:11:35.436 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:35.437 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:11:35.437 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019565+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:08.019565+0000', 'last_peered': '2023-11-24T09:11:08.019565+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:08.019565+0000', 'last_undegraded': '2023-11-24T09:11:08.019565+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:35.437 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:11:35.437 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 654, 'reported_epoch': 503, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:08.015143+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:08.015143+0000', 'last_peered': '2023-11-24T09:11:08.015143+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:08.015143+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:35.437 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:11:35.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~218s 2023-11-24T09:11:35.754 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:35.754 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2531,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-11-24T09:11:18.928467+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-11-24T09:11:35.771 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:11:35.771 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:36.036 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:36.036 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:36.053 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1762,"stamp":"2023-11-24T09:11:34.934218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004353"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738877,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.625}]}]},{"osd":1,"up_from":8,"seq":34359738876,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.53600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:39.055 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:39.352 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:39.353 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:39.370 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1764,"stamp":"2023-11-24T09:11:38.935795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738878,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.52800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.53600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:39.371 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:39.657 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:39.658 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:39.675 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1764,"stamp":"2023-11-24T09:11:38.935795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738878,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.52800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.53600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:39.676 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:11:39.676 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:39.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:11:39.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:39.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:11:39.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:39.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:11:39.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:39.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:11:39.677 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019565+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:08.019565+0000', 'last_peered': '2023-11-24T09:11:08.019565+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:08.019565+0000', 'last_undegraded': '2023-11-24T09:11:08.019565+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:39.677 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:11:39.678 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 654, 'reported_epoch': 503, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:08.015143+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:08.015143+0000', 'last_peered': '2023-11-24T09:11:08.015143+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:08.015143+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:39.678 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:11:40.011 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:40.011 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2535,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-11-24T09:11:18.928467+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-11-24T09:11:40.030 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:11:40.030 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:40.300 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:40.300 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:40.317 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1764,"stamp":"2023-11-24T09:11:38.935795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738878,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.52800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.53600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:40.996 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~224s 2023-11-24T09:11:43.319 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:43.620 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:43.638 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1766,"stamp":"2023-11-24T09:11:42.937264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004501"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.40400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.53600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:43.639 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:43.928 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:43.928 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:43.946 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1766,"stamp":"2023-11-24T09:11:42.937264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004501"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.40400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.53600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:43.947 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:11:43.947 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:43.947 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:11:43.947 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:43.947 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:11:43.947 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:43.947 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:11:43.947 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:43.948 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:11:43.948 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019565+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:08.019565+0000', 'last_peered': '2023-11-24T09:11:08.019565+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:08.019565+0000', 'last_undegraded': '2023-11-24T09:11:08.019565+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:43.948 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:11:43.948 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 654, 'reported_epoch': 503, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:08.015143+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:08.015143+0000', 'last_peered': '2023-11-24T09:11:08.015143+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:08.015143+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:43.948 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:11:44.279 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:44.280 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2539,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":51,"modified":"2023-11-24T09:11:18.928467+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-11-24T09:11:44.296 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:11:44.296 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:44.583 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:44.583 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:44.601 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1766,"stamp":"2023-11-24T09:11:42.937264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004501"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":519,"reported_epoch":503,"state":"active+clean","last_fresh":"2023-11-24T09:11:08.110303+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:08.110303+0000","last_peered":"2023-11-24T09:11:08.110303+0000","last_clean":"2023-11-24T09:11:08.110303+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:08.110303+0000","last_undegraded":"2023-11-24T09:11:08.110303+0000","last_fullsized":"2023-11-24T09:11:08.110303+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":654,"reported_epoch":503,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:08.015143+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:08.015143+0000","last_peered":"2023-11-24T09:11:08.015143+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:08.015143+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738879,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.40400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.53600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:46.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~229s 2023-11-24T09:11:52.824 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:53.127 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:53.128 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:53.146 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1771,"stamp":"2023-11-24T09:11:52.940852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004285"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.59899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:53.147 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:53.430 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:53.430 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:53.448 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1771,"stamp":"2023-11-24T09:11:52.940852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004285"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.59899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:53.449 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:11:53.449 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019565+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:08.019565+0000', 'last_peered': '2023-11-24T09:11:08.019565+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:08.019565+0000', 'last_undegraded': '2023-11-24T09:11:08.019565+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:11:53.450 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:53.451 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:11:53.783 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:53.783 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2549,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:11:53.800 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:11:53.800 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:54.079 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1771,"stamp":"2023-11-24T09:11:52.940852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004285"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:10:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.497,"15min":0.58799999999999997},"min":{"1min":0.218,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.58099999999999996,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.32300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.6719999999999999,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.59899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:57.082 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:57.377 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:57.378 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:57.395 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1773,"stamp":"2023-11-24T09:11:56.942329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004366"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738880,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.59699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:57.396 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:57.680 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:57.680 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:57.698 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1773,"stamp":"2023-11-24T09:11:56.942329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004366"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738880,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.59699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:11:57.699 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:11:57.699 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:57.699 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:11:57.699 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:57.699 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:11:57.699 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:57.700 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:11:57.700 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:57.700 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:11:57.700 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019565+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:08.019565+0000', 'last_peered': '2023-11-24T09:11:08.019565+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:08.019565+0000', 'last_undegraded': '2023-11-24T09:11:08.019565+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:57.700 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:11:57.700 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:11:57.700 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:11:58.043 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:58.043 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2553,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:11:58.062 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:11:58.063 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:11:58.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~241s 2023-11-24T09:11:58.339 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:11:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:11:58.355 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1773,"stamp":"2023-11-24T09:11:56.942329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004366"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738880,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.59699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:01.357 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:01.652 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:01.653 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:01.670 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1775,"stamp":"2023-11-24T09:12:00.943785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004356"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738882,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.69199999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738881,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.375}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:01.671 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:01.949 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:01.949 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:01.967 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1775,"stamp":"2023-11-24T09:12:00.943785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004356"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738882,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.69199999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738881,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.375}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:01.968 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:01.968 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:01.968 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:01.968 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:01.968 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:01.968 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:01.968 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:01.969 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:01.969 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:01.969 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 23, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019565+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:08.019565+0000', 'last_peered': '2023-11-24T09:11:08.019565+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:08.019565+0000', 'last_undegraded': '2023-11-24T09:11:08.019565+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:01.969 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:01.969 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:01.969 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:02.299 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:02.299 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2557,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:02.317 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:02.317 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:02.595 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:02.595 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:02.611 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1775,"stamp":"2023-11-24T09:12:00.943785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004356"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019565+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:08.019565+0000","last_peered":"2023-11-24T09:11:08.019565+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:08.019565+0000","last_undegraded":"2023-11-24T09:11:08.019565+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738882,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:10:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.59599999999999997},"min":{"1min":0.39400000000000002,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":1.143,"5min":17.273,"15min":17.273},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.167,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.69199999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738881,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.375}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:03.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~246s 2023-11-24T09:12:05.613 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:05.902 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:05.903 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:05.919 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1777,"stamp":"2023-11-24T09:12:04.945095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004242"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.66500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738882,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:05.920 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:06.186 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:06.187 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:06.203 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1777,"stamp":"2023-11-24T09:12:04.945095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004242"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.66500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738882,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:06.204 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:06.204 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:06.205 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:06.205 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:06.205 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:06.205 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:06.206 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:06.206 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:06.206 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:06.206 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:06.207 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:06.207 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:06.207 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:06.538 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:06.539 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2562,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:06.556 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:06.557 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:06.838 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:06.838 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:06.856 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1777,"stamp":"2023-11-24T09:12:04.945095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004242"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.66500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738882,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:09.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~252s 2023-11-24T09:12:09.857 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:10.156 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:10.157 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:10.175 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1779,"stamp":"2023-11-24T09:12:08.946207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003878"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738884,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.498},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.65800000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.48199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:10.176 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:10.506 INFO:teuthology.orchestra.run.smithi164.stderr:2023-11-24T09:12:10.505+0000 7fb605fe6640 0 --2- 172.21.15.164:0/4135229602 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6000b0e20 0x7fb60013a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-24T09:12:10.697 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:10.698 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:10.716 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1779,"stamp":"2023-11-24T09:12:08.946207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003878"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738884,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.498},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.65800000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.48199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:10.717 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:10.717 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:10.717 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:10.717 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:10.717 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:10.717 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:10.717 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:10.718 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:10.718 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:10.718 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:10.718 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:10.718 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:10.718 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:11.049 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:11.050 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2566,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:11.066 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:11.067 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:11.354 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:11.373 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1780,"stamp":"2023-11-24T09:12:10.946819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003795"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738884,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.498},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.65800000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.48199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:14.375 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:14.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~257s 2023-11-24T09:12:14.672 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:14.673 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:14.691 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1781,"stamp":"2023-11-24T09:12:12.947378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.437},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.55500000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.48199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:14.692 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:14.965 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:14.966 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:14.982 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1782,"stamp":"2023-11-24T09:12:14.947964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003466"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.437},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.55500000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738884,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.59799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:14.983 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:14.983 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:14.984 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:14.984 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:14.984 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:14.984 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:14.984 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:14.984 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:14.984 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:14.984 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:14.984 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:14.985 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:14.985 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:15.305 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:15.305 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2570,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:15.324 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:15.324 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:15.607 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:15.607 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:15.623 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1782,"stamp":"2023-11-24T09:12:14.947964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003466"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.437},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.55500000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738884,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.59799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:18.625 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:18.913 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:18.914 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:18.930 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1783,"stamp":"2023-11-24T09:12:16.948534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003438"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.437},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.55500000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738884,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.59799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:18.931 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:19.222 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:19.241 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1784,"stamp":"2023-11-24T09:12:18.949124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003428"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.52400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:19.242 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:19.242 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:19.242 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:19.242 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:19.242 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:19.242 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:19.242 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:19.242 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:19.243 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:19.243 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:19.243 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:19.243 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:19.243 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:19.561 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:19.561 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2575,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:19.578 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:19.579 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:19.840 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:19.840 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:19.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~263s 2023-11-24T09:12:19.857 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1784,"stamp":"2023-11-24T09:12:18.949124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003428"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.52400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:22.858 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:23.159 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:23.160 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:23.177 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1786,"stamp":"2023-11-24T09:12:22.950371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:23.178 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:23.466 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:23.466 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:23.484 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1786,"stamp":"2023-11-24T09:12:22.950371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:23.485 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:23.485 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:23.485 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:23.485 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:23.486 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:23.486 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:23.486 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:23.486 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:23.486 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:23.486 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:23.486 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:23.486 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:23.487 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:23.821 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:23.821 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2579,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:23.839 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:23.839 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:24.103 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:24.103 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:24.120 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1786,"stamp":"2023-11-24T09:12:22.950371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:25.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~268s 2023-11-24T09:12:27.121 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:27.420 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:27.420 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:27.438 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1788,"stamp":"2023-11-24T09:12:26.951507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003543"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738886,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:27.439 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:27.713 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:27.730 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1788,"stamp":"2023-11-24T09:12:26.951507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003543"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738886,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:27.731 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:27.732 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:27.732 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:27.732 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:27.732 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:27.732 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:27.732 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:27.732 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:27.732 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:27.732 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:27.733 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:27.733 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:27.733 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:28.079 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:28.079 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2583,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:28.099 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:28.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:28.375 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:28.376 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:28.392 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1788,"stamp":"2023-11-24T09:12:26.951507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003543"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738886,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.52300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:30.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~273s 2023-11-24T09:12:31.394 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:31.691 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:31.708 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1790,"stamp":"2023-11-24T09:12:30.952622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003498"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.34799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.51700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:31.709 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:31.975 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:31.976 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:31.992 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1790,"stamp":"2023-11-24T09:12:30.952622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003498"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.34799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.51700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:31.992 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:31.993 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:31.993 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:31.993 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:31.993 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:31.993 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:31.993 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:31.993 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:31.993 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:31.993 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:31.994 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:31.994 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:31.994 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:32.340 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:32.340 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2587,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:32.359 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:32.359 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:32.622 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:32.622 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:32.639 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1790,"stamp":"2023-11-24T09:12:30.952622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003498"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.34799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.51700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:35.640 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:35.934 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:35.934 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:35.952 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1792,"stamp":"2023-11-24T09:12:34.953714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.63200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738888,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:35.953 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:36.054 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~279s 2023-11-24T09:12:36.239 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:36.239 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:36.258 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1792,"stamp":"2023-11-24T09:12:34.953714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.63200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738888,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:36.259 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:36.259 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:36.259 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:36.259 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:36.259 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:36.259 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:36.259 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:36.259 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:36.260 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:36.260 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:36.260 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:36.260 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:36.260 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:36.585 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:36.586 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2592,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:36.603 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:36.603 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:36.888 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:36.888 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:36.905 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1792,"stamp":"2023-11-24T09:12:34.953714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738889,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.63200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738888,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.41199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:39.907 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:40.213 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:40.213 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:40.231 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1794,"stamp":"2023-11-24T09:12:38.954952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.65400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.438}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:40.232 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:40.516 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:40.517 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:40.534 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1794,"stamp":"2023-11-24T09:12:38.954952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.65400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.438}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:40.535 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:40.535 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:40.535 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:40.535 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:40.535 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:40.535 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:40.535 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:40.535 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:40.536 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:40.536 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:40.536 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:40.536 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:40.536 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:40.881 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:40.882 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2596,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:40.900 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:40.900 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:41.164 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:41.185 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:41.186 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1795,"stamp":"2023-11-24T09:12:40.955579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738890,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.65400000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.438}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:41.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~284s 2023-11-24T09:12:44.188 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:44.475 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:44.476 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:44.492 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1796,"stamp":"2023-11-24T09:12:42.956169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003546"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.59999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.438}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:44.493 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:44.763 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:44.763 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:44.779 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1796,"stamp":"2023-11-24T09:12:42.956169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003546"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.59999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.438}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:44.780 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:44.780 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:44.780 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:44.780 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:44.780 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:44.780 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:44.781 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:44.781 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:44.781 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:44.781 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:44.781 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:44.781 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:44.781 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:45.122 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:45.122 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2600,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:45.142 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:45.142 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:45.414 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:45.432 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1797,"stamp":"2023-11-24T09:12:44.956756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003605"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.59999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.47199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:46.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~290s 2023-11-24T09:12:48.434 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:48.721 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:48.722 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:48.739 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1798,"stamp":"2023-11-24T09:12:46.957365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003651"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.59999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.47199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:48.741 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:49.010 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:49.010 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:49.026 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1799,"stamp":"2023-11-24T09:12:48.957934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003571"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.65100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.50800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:49.027 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:49.027 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:49.028 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:49.028 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:49.028 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:49.028 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:49.028 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:49.028 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:49.028 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:49.028 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:49.028 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:49.029 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:49.029 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:49.351 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:49.352 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2604,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:49.369 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:49.369 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:49.646 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:49.647 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:49.663 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1799,"stamp":"2023-11-24T09:12:48.957934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003571"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.65100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.50800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:52.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~295s 2023-11-24T09:12:52.665 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:52.952 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:52.952 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:52.970 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1800,"stamp":"2023-11-24T09:12:50.958602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003649"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.65100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.50800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:52.971 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:53.256 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:53.256 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:53.274 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1801,"stamp":"2023-11-24T09:12:52.959186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003606"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.70899999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.50800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:53.275 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:53.275 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:53.275 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:53.275 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:53.275 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:53.276 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:53.276 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:53.276 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:53.276 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:53.276 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:53.276 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:53.276 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:53.276 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:53.615 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:53.615 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2609,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:53.632 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:53.633 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:53.917 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:53.936 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1801,"stamp":"2023-11-24T09:12:52.959186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003606"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.70899999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:11:50 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.58299999999999996},"min":{"1min":0.33000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.74099999999999999,"5min":2.4199999999999999,"15min":10.954000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.26900000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.873,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.50800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:56.938 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:57.236 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:57.236 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:57.255 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1803,"stamp":"2023-11-24T09:12:56.960375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003619"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.70899999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738892,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45000000000000001,"15min":0.55700000000000005},"min":{"1min":0.27800000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.56999999999999995,"5min":0.74099999999999999,"15min":10.954000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.311,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.627,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.56499999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:57.256 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:57.523 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:57.524 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2023-11-24T09:12:57.540 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1803,"stamp":"2023-11-24T09:12:56.960375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3006500,"kb_used_data":419520,"kb_used_omap":292,"kb_used_meta":2586651,"kb_avail":280109020,"statfs":{"total":289910292480,"available":286831636480,"internally_reserved":0,"allocated":429588480,"data_stored":420652373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":299488,"internal_metadata":2648731168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003619"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.011204+0000","last_change":"2023-11-24T09:07:54.755229+0000","last_active":"2023-11-24T09:11:08.011204+0000","last_peered":"2023-11-24T09:11:08.011204+0000","last_clean":"2023-11-24T09:07:52.681788+0000","last_became_active":"2023-11-24T09:07:54.755229+0000","last_became_peered":"2023-11-24T09:07:54.755229+0000","last_unstale":"2023-11-24T09:11:08.011204+0000","last_undegraded":"2023-11-24T09:11:08.011204+0000","last_fullsized":"2023-11-24T09:07:54.714529+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":523,"reported_epoch":503,"state":"active+undersized","last_fresh":"2023-11-24T09:11:08.019566+0000","last_change":"2023-11-24T09:07:54.766490+0000","last_active":"2023-11-24T09:11:08.019566+0000","last_peered":"2023-11-24T09:11:08.019566+0000","last_clean":"2023-11-24T09:07:52.681597+0000","last_became_active":"2023-11-24T09:07:54.766490+0000","last_became_peered":"2023-11-24T09:07:54.766490+0000","last_unstale":"2023-11-24T09:11:08.019566+0000","last_undegraded":"2023-11-24T09:11:08.019566+0000","last_fullsized":"2023-11-24T09:07:54.711733+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.822041+0000","last_change":"2023-11-24T09:07:54.771715+0000","last_active":"2023-11-24T09:11:13.822041+0000","last_peered":"2023-11-24T09:11:13.822041+0000","last_clean":"2023-11-24T09:07:52.683928+0000","last_became_active":"2023-11-24T09:07:54.771715+0000","last_became_peered":"2023-11-24T09:07:54.771715+0000","last_unstale":"2023-11-24T09:11:13.822041+0000","last_undegraded":"2023-11-24T09:11:13.822041+0000","last_fullsized":"2023-11-24T09:07:54.712018+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.817433+0000","last_change":"2023-11-24T09:07:54.723124+0000","last_active":"2023-11-24T09:11:13.817433+0000","last_peered":"2023-11-24T09:11:13.817433+0000","last_clean":"2023-11-24T09:07:52.680777+0000","last_became_active":"2023-11-24T09:07:54.723124+0000","last_became_peered":"2023-11-24T09:07:54.723124+0000","last_unstale":"2023-11-24T09:11:13.817433+0000","last_undegraded":"2023-11-24T09:11:13.817433+0000","last_fullsized":"2023-11-24T09:07:54.701536+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":504,"state":"active+undersized","last_fresh":"2023-11-24T09:11:13.818492+0000","last_change":"2023-11-24T09:07:54.774978+0000","last_active":"2023-11-24T09:11:13.818492+0000","last_peered":"2023-11-24T09:11:13.818492+0000","last_clean":"2023-11-24T08:29:40.519173+0000","last_became_active":"2023-11-24T09:07:54.774978+0000","last_became_peered":"2023-11-24T09:07:54.774978+0000","last_unstale":"2023-11-24T09:11:13.818492+0000","last_undegraded":"2023-11-24T09:11:13.818492+0000","last_fullsized":"2023-11-24T09:07:54.710134+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":519,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.627415+0000","last_change":"2023-11-24T08:29:41.541315+0000","last_active":"2023-11-24T09:11:12.627415+0000","last_peered":"2023-11-24T09:11:12.627415+0000","last_clean":"2023-11-24T09:11:12.627415+0000","last_became_active":"2023-11-24T08:29:41.540994+0000","last_became_peered":"2023-11-24T08:29:41.540994+0000","last_unstale":"2023-11-24T09:11:12.627415+0000","last_undegraded":"2023-11-24T09:11:12.627415+0000","last_fullsized":"2023-11-24T09:11:12.627415+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:06:53.556619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:12.628328+0000","last_change":"2023-11-24T08:29:41.542963+0000","last_active":"2023-11-24T09:11:12.628328+0000","last_peered":"2023-11-24T09:11:12.628328+0000","last_clean":"2023-11-24T09:11:12.628328+0000","last_became_active":"2023-11-24T08:29:41.542656+0000","last_became_peered":"2023-11-24T08:29:41.542656+0000","last_unstale":"2023-11-24T09:11:12.628328+0000","last_undegraded":"2023-11-24T09:11:12.628328+0000","last_fullsized":"2023-11-24T09:11:12.628328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:05:19.985615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":520,"reported_epoch":504,"state":"active+clean","last_fresh":"2023-11-24T09:11:13.819444+0000","last_change":"2023-11-24T08:29:41.540641+0000","last_active":"2023-11-24T09:11:13.819444+0000","last_peered":"2023-11-24T09:11:13.819444+0000","last_clean":"2023-11-24T09:11:13.819444+0000","last_became_active":"2023-11-24T08:29:41.540333+0000","last_became_peered":"2023-11-24T08:29:41.540333+0000","last_unstale":"2023-11-24T09:11:13.819444+0000","last_undegraded":"2023-11-24T09:11:13.819444+0000","last_fullsized":"2023-11-24T09:11:13.819444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:40.519173+0000","last_clean_scrub_stamp":"2023-11-24T08:29:40.519173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:03:23.632604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":655,"reported_epoch":504,"state":"active+undersized+degraded","last_fresh":"2023-11-24T09:11:13.821823+0000","last_change":"2023-11-24T09:07:54.756445+0000","last_active":"2023-11-24T09:11:13.821823+0000","last_peered":"2023-11-24T09:11:13.821823+0000","last_clean":"2023-11-24T09:07:52.679980+0000","last_became_active":"2023-11-24T09:07:54.756445+0000","last_became_peered":"2023-11-24T09:07:54.756445+0000","last_unstale":"2023-11-24T09:11:13.821823+0000","last_undegraded":"2023-11-24T09:07:54.705697+0000","last_fullsized":"2023-11-24T09:07:54.705415+0000","mapping_epoch":489,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:36.438275+0000","last_clean_scrub_stamp":"2023-11-24T08:29:36.438275+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738893,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":771432,"kb_used_data":5852,"kb_used_omap":97,"kb_used_meta":765470,"kb_avail":93600408,"statfs":{"total":96636764160,"available":95846817792,"internally_reserved":0,"allocated":5992448,"data_stored":2989602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99711,"internal_metadata":783841921},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:06:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51100000000000001,"15min":0.51400000000000001},"min":{"1min":0.21099999999999999,"5min":0.159,"15min":0.159},"max":{"1min":0.60799999999999998,"5min":3.3540000000000001,"15min":3.3540000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.53000000000000003},"min":{"1min":0.19500000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.64700000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.70299999999999996,"15min":0.59099999999999997},"min":{"1min":0.252,"5min":0.16500000000000001,"15min":0.16500000000000001},"max":{"1min":0.61099999999999999,"5min":17.273,"15min":17.273},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.79600000000000004,"15min":0.69199999999999995},"min":{"1min":0.29599999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.69199999999999995,"5min":17.606000000000002,"15min":17.606000000000002},"last":0.70899999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738892,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":840300,"kb_used_data":6432,"kb_used_omap":93,"kb_used_meta":833762,"kb_avail":93531540,"statfs":{"total":96636764160,"available":95776296960,"internally_reserved":0,"allocated":6586368,"data_stored":3579970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95922,"internal_metadata":853772622},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:07:42 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.64800000000000002,"15min":0.60199999999999998},"min":{"1min":0.25800000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.2090000000000001,"5min":4.4169999999999998,"15min":10.991},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.25,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.089,"5min":2.6850000000000001,"15min":6.5780000000000003},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45000000000000001,"15min":0.55700000000000005},"min":{"1min":0.27800000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.56999999999999995,"5min":0.74099999999999999,"15min":10.954000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.311,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.627,"5min":6.4000000000000004,"15min":6.4000000000000004},"last":0.56499999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738831,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1394768,"kb_used_data":407236,"kb_used_omap":101,"kb_used_meta":987418,"kb_avail":92977072,"statfs":{"total":96636764160,"available":95208521728,"internally_reserved":0,"allocated":417009664,"data_stored":414082801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103855,"internal_metadata":1011116625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.69099999999999995,"15min":0.59399999999999997},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17000000000000001},"max":{"1min":0.627,"5min":7.7050000000000001,"15min":9.0519999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.69999999999999996,"15min":0.623},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.73999999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Nov 24 09:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.75600000000000001,"15min":0.65800000000000003},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.70699999999999996,"5min":7.7489999999999997,"15min":9.0039999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.85999999999999999,"15min":0.76500000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.79700000000000004,"5min":7.8029999999999999,"15min":9.2409999999999997},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:12:57.541 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-24T09:12:57.541 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.011204+0000', 'last_change': '2023-11-24T09:07:54.755229+0000', 'last_active': '2023-11-24T09:11:08.011204+0000', 'last_peered': '2023-11-24T09:11:08.011204+0000', 'last_clean': '2023-11-24T09:07:52.681788+0000', 'last_became_active': '2023-11-24T09:07:54.755229+0000', 'last_became_peered': '2023-11-24T09:07:54.755229+0000', 'last_unstale': '2023-11-24T09:11:08.011204+0000', 'last_undegraded': '2023-11-24T09:11:08.011204+0000', 'last_fullsized': '2023-11-24T09:07:54.714529+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:01:02.162392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:57.541 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-24T09:12:57.541 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 523, 'reported_epoch': 503, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:08.019566+0000', 'last_change': '2023-11-24T09:07:54.766490+0000', 'last_active': '2023-11-24T09:11:08.019566+0000', 'last_peered': '2023-11-24T09:11:08.019566+0000', 'last_clean': '2023-11-24T09:07:52.681597+0000', 'last_became_active': '2023-11-24T09:07:54.766490+0000', 'last_became_peered': '2023-11-24T09:07:54.766490+0000', 'last_unstale': '2023-11-24T09:11:08.019566+0000', 'last_undegraded': '2023-11-24T09:11:08.019566+0000', 'last_fullsized': '2023-11-24T09:07:54.711733+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T17:17:30.589411+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:57.541 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-24T09:12:57.542 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.822041+0000', 'last_change': '2023-11-24T09:07:54.771715+0000', 'last_active': '2023-11-24T09:11:13.822041+0000', 'last_peered': '2023-11-24T09:11:13.822041+0000', 'last_clean': '2023-11-24T09:07:52.683928+0000', 'last_became_active': '2023-11-24T09:07:54.771715+0000', 'last_became_peered': '2023-11-24T09:07:54.771715+0000', 'last_unstale': '2023-11-24T09:11:13.822041+0000', 'last_undegraded': '2023-11-24T09:11:13.822041+0000', 'last_fullsized': '2023-11-24T09:07:54.712018+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:43:55.755277+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:57.542 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-24T09:12:57.542 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 524, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.817433+0000', 'last_change': '2023-11-24T09:07:54.723124+0000', 'last_active': '2023-11-24T09:11:13.817433+0000', 'last_peered': '2023-11-24T09:11:13.817433+0000', 'last_clean': '2023-11-24T09:07:52.680777+0000', 'last_became_active': '2023-11-24T09:07:54.723124+0000', 'last_became_peered': '2023-11-24T09:07:54.723124+0000', 'last_unstale': '2023-11-24T09:11:13.817433+0000', 'last_undegraded': '2023-11-24T09:11:13.817433+0000', 'last_fullsized': '2023-11-24T09:07:54.701536+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T18:48:07.367991+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:57.542 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-24T09:12:57.542 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 24, 'reported_epoch': 504, 'state': 'active+undersized', 'last_fresh': '2023-11-24T09:11:13.818492+0000', 'last_change': '2023-11-24T09:07:54.774978+0000', 'last_active': '2023-11-24T09:11:13.818492+0000', 'last_peered': '2023-11-24T09:11:13.818492+0000', 'last_clean': '2023-11-24T08:29:40.519173+0000', 'last_became_active': '2023-11-24T09:07:54.774978+0000', 'last_became_peered': '2023-11-24T09:07:54.774978+0000', 'last_unstale': '2023-11-24T09:11:13.818492+0000', 'last_undegraded': '2023-11-24T09:11:13.818492+0000', 'last_fullsized': '2023-11-24T09:07:54.710134+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:40.519173+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T16:21:06.168459+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:57.542 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-24T09:12:57.542 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 655, 'reported_epoch': 504, 'state': 'active+undersized+degraded', 'last_fresh': '2023-11-24T09:11:13.821823+0000', 'last_change': '2023-11-24T09:07:54.756445+0000', 'last_active': '2023-11-24T09:11:13.821823+0000', 'last_peered': '2023-11-24T09:11:13.821823+0000', 'last_clean': '2023-11-24T09:07:52.679980+0000', 'last_became_active': '2023-11-24T09:07:54.756445+0000', 'last_became_peered': '2023-11-24T09:07:54.756445+0000', 'last_unstale': '2023-11-24T09:11:13.821823+0000', 'last_undegraded': '2023-11-24T09:07:54.705697+0000', 'last_fullsized': '2023-11-24T09:07:54.705415+0000', 'mapping_epoch': 489, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'last_clean_scrub_stamp': '2023-11-24T08:29:36.438275+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-11-25T19:45:29.704812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-24T09:12:57.542 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-24T09:12:57.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~301s 2023-11-24T09:12:57.667 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-24T09:12:57.851 INFO:teuthology.orchestra.run.smithi164.stdout: 2023-11-24T09:12:57.852 INFO:teuthology.orchestra.run.smithi164.stdout:{"fsid":"6f0e426c-18ac-4eb5-9ad7-dc680622a77c","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2613,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":504,"num_osds":3,"num_up_osds":2,"osd_up_since":1700816873,"num_in_osds":3,"osd_in_since":1700814565,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3078656000,"bytes_avail":286831636480,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":52,"modified":"2023-11-24T09:11:50.940183+0000","services":{}},"progress_events":{}} 2023-11-24T09:12:57.869 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-24T09:12:57.869 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-24T09:12:58.069 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-11-24T09:12:58.070 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-11-24T09:12:58.070 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-11-24T09:12:58.070 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-11-24T09:12:58.071 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T09:12:58.068+0000 7fee3d47e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 34831) UID: 0 2023-11-24T09:12:58.071 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T09:12:58.068+0000 7fee3d47e640 -1 osd.1 504 *** Got signal Terminated *** 2023-11-24T09:12:58.071 INFO:tasks.ceph.cluster2.osd.1.smithi164.stderr:2023-11-24T09:12:58.068+0000 7fee3d47e640 -1 osd.1 504 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-24T09:12:58.071 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:12:58.068+0000 7f42be3c6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 34833) UID: 0 2023-11-24T09:12:58.072 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:12:58.068+0000 7f42be3c6640 -1 osd.2 504 *** Got signal Terminated *** 2023-11-24T09:12:58.072 INFO:tasks.ceph.cluster2.osd.2.smithi164.stderr:2023-11-24T09:12:58.068+0000 7f42be3c6640 -1 osd.2 504 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-24T09:12:58.072 INFO:tasks.ceph.cluster2.mon.a.smithi164.stderr:2023-11-24T09:12:58.068+0000 7f7d25521640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 34683) UID: 0 2023-11-24T09:12:58.072 INFO:tasks.ceph.cluster2.mon.a.smithi164.stderr:2023-11-24T09:12:58.068+0000 7f7d25521640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-24T09:12:58.072 INFO:tasks.ceph.cluster2.mgr.x.smithi153.stderr:2023-11-24T09:12:58.068+0000 7fd961a0a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 39518) UID: 0 2023-11-24T09:14:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-24T09:14:57.889 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2771, in wait_for_clean cur_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-24T09:14:57.891 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-24T09:14:57.891 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-24T09:14:57.891 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-11-24T09:14:57.892 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:14:57.892 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-24T09:14:57.892 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2771, in wait_for_clean cur_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-11-24T09:14:57.892 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-11-24T09:14:57.893 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-11-24T09:14:57.893 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:14:57.893 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-11-24T09:14:57.893 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-11-24T09:14:57.893 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:14:57.893 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-11-24T09:14:57.893 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-24T09:14:57.894 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-11-24T09:14:57.894 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:14:57.894 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-11-24T09:14:57.894 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-24T09:14:57.894 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-11-24T09:14:57.894 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:14:57.894 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-11-24T09:14:57.894 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-24T09:14:57.895 DEBUG:teuthology.orchestra.run.smithi164:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-11-24T09:14:57.909 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T09:14:57.910 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-11-24T09:14:58.246 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T09:14:58.246 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-11-24T09:14:58.509 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T09:14:58.509 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-11-24T09:14:58.855 INFO:tasks.ceph:Archiving mon data... 2023-11-24T09:14:58.856 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501/data/cluster2.mon.a.tgz 2023-11-24T09:14:58.856 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2023-11-24T09:14:58.861 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.3bUIPqyKwQ 2023-11-24T09:14:58.861 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.3bUIPqyKwQ 2023-11-24T09:14:58.999 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.3bUIPqyKwQ 2023-11-24T09:14:59.064 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.3bUIPqyKwQ is 421KB 2023-11-24T09:14:59.124 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.3bUIPqyKwQ 2023-11-24T09:14:59.130 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-24T09:14:59.130 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-11-24T09:14:59.133 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-11-24T09:14:59.187 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2771, in wait_for_clean cur_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-24T09:14:59.187 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-24T09:14:59.200 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-11-24T09:14:59.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-24T09:14:59.495 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T09:14:59.495 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-24T09:14:59.511 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":1850,"stamp":"2023-11-24T09:14:57.585134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":493240,"kb_used_data":12560,"kb_used_omap":4,"kb_used_meta":480635,"kb_avail":282622280,"statfs":{"total":289910292480,"available":289405214720,"internally_reserved":0,"allocated":12861440,"data_stored":7438583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":492170575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003632"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:23.988167+0000","last_change":"2023-11-24T08:29:04.820862+0000","last_active":"2023-11-24T09:12:23.988167+0000","last_peered":"2023-11-24T09:12:23.988167+0000","last_clean":"2023-11-24T09:12:23.988167+0000","last_became_active":"2023-11-24T08:29:04.818386+0000","last_became_peered":"2023-11-24T08:29:04.818386+0000","last_unstale":"2023-11-24T09:12:23.988167+0000","last_undegraded":"2023-11-24T09:12:23.988167+0000","last_fullsized":"2023-11-24T09:12:23.988167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:44:06.511981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:23.988123+0000","last_change":"2023-11-24T08:29:04.818405+0000","last_active":"2023-11-24T09:12:23.988123+0000","last_peered":"2023-11-24T09:12:23.988123+0000","last_clean":"2023-11-24T09:12:23.988123+0000","last_became_active":"2023-11-24T08:29:04.817967+0000","last_became_peered":"2023-11-24T08:29:04.817967+0000","last_unstale":"2023-11-24T09:12:23.988123+0000","last_undegraded":"2023-11-24T09:12:23.988123+0000","last_fullsized":"2023-11-24T09:12:23.988123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:51.528140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.143584+0000","last_change":"2023-11-24T08:29:04.819596+0000","last_active":"2023-11-24T09:11:10.143584+0000","last_peered":"2023-11-24T09:11:10.143584+0000","last_clean":"2023-11-24T09:11:10.143584+0000","last_became_active":"2023-11-24T08:29:04.818295+0000","last_became_peered":"2023-11-24T08:29:04.818295+0000","last_unstale":"2023-11-24T09:11:10.143584+0000","last_undegraded":"2023-11-24T09:11:10.143584+0000","last_fullsized":"2023-11-24T09:11:10.143584+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:22:22.642040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.138101+0000","last_change":"2023-11-24T08:29:04.820593+0000","last_active":"2023-11-24T09:11:10.138101+0000","last_peered":"2023-11-24T09:11:10.138101+0000","last_clean":"2023-11-24T09:11:10.138101+0000","last_became_active":"2023-11-24T08:29:04.818750+0000","last_became_peered":"2023-11-24T08:29:04.818750+0000","last_unstale":"2023-11-24T09:11:10.138101+0000","last_undegraded":"2023-11-24T09:11:10.138101+0000","last_fullsized":"2023-11-24T09:11:10.138101+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:17:33.495447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:49.272528+0000","last_change":"2023-11-24T08:29:04.820326+0000","last_active":"2023-11-24T09:12:49.272528+0000","last_peered":"2023-11-24T09:12:49.272528+0000","last_clean":"2023-11-24T09:12:49.272528+0000","last_became_active":"2023-11-24T08:29:04.819285+0000","last_became_peered":"2023-11-24T08:29:04.819285+0000","last_unstale":"2023-11-24T09:12:49.272528+0000","last_undegraded":"2023-11-24T09:12:49.272528+0000","last_fullsized":"2023-11-24T09:12:49.272528+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:42:03.353215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":492,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:48.885181+0000","last_change":"2023-11-24T08:29:04.819744+0000","last_active":"2023-11-24T09:12:48.885181+0000","last_peered":"2023-11-24T09:12:48.885181+0000","last_clean":"2023-11-24T09:12:48.885181+0000","last_became_active":"2023-11-24T08:29:04.819191+0000","last_became_peered":"2023-11-24T08:29:04.819191+0000","last_unstale":"2023-11-24T09:12:48.885181+0000","last_undegraded":"2023-11-24T09:12:48.885181+0000","last_fullsized":"2023-11-24T09:12:48.885181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:09:02.746576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":492,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:18.872164+0000","last_change":"2023-11-24T08:29:04.821790+0000","last_active":"2023-11-24T09:11:18.872164+0000","last_peered":"2023-11-24T09:11:18.872164+0000","last_clean":"2023-11-24T09:11:18.872164+0000","last_became_active":"2023-11-24T08:29:04.821369+0000","last_became_peered":"2023-11-24T08:29:04.821369+0000","last_unstale":"2023-11-24T09:11:18.872164+0000","last_undegraded":"2023-11-24T09:11:18.872164+0000","last_fullsized":"2023-11-24T09:11:18.872164+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:52:15.752249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.140828+0000","last_change":"2023-11-24T08:29:04.820777+0000","last_active":"2023-11-24T09:11:10.140828+0000","last_peered":"2023-11-24T09:11:10.140828+0000","last_clean":"2023-11-24T09:11:10.140828+0000","last_became_active":"2023-11-24T08:29:04.817968+0000","last_became_peered":"2023-11-24T08:29:04.817968+0000","last_unstale":"2023-11-24T09:11:10.140828+0000","last_undegraded":"2023-11-24T09:11:10.140828+0000","last_fullsized":"2023-11-24T09:11:10.140828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:03:59.754900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":621,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.143030+0000","last_change":"2023-11-24T08:28:59.784048+0000","last_active":"2023-11-24T09:11:10.143030+0000","last_peered":"2023-11-24T09:11:10.143030+0000","last_clean":"2023-11-24T09:11:10.143030+0000","last_became_active":"2023-11-24T08:28:59.783498+0000","last_became_peered":"2023-11-24T08:28:59.783498+0000","last_unstale":"2023-11-24T09:11:10.143030+0000","last_undegraded":"2023-11-24T09:11:10.143030+0000","last_fullsized":"2023-11-24T09:11:10.143030+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_clean_scrub_stamp":"2023-11-24T08:28:58.760510+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:12:39.581115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738924,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165412,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206428,"statfs":{"total":96636764160,"available":96467382272,"internally_reserved":0,"allocated":3891200,"data_stored":2085949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:14:16 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.77300000000000002,"15min":0.625},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.17399999999999999},"max":{"1min":0.69599999999999995,"5min":25.623000000000001,"15min":25.623000000000001},"last":0.624},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.90700000000000003,"15min":0.755},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.246},"max":{"1min":0.77600000000000002,"5min":25.969999999999999,"15min":25.969999999999999},"last":0.753}]},{"osd":1,"last update":"Fri Nov 24 09:14:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54800000000000004,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.20399999999999999},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":4.7060000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.51300000000000001,"5min":0.50700000000000001,"15min":0.23599999999999999},"max":{"1min":0.69699999999999995,"5min":4.2270000000000003,"15min":4.7549999999999999},"last":0.55500000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738924,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152104,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94219736,"statfs":{"total":96636764160,"available":96481009664,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52900000000000003,"15min":0.53800000000000003},"min":{"1min":0.46200000000000002,"5min":0.432,"15min":0.17699999999999999},"max":{"1min":0.61599999999999999,"5min":0.69099999999999995,"15min":4.2889999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.53200000000000003,"5min":0.41199999999999998,"15min":0.20200000000000001},"max":{"1min":0.69099999999999995,"5min":0.78900000000000003,"15min":4.4130000000000003},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.19800000000000001},"max":{"1min":0.72599999999999998,"5min":0.92600000000000005,"15min":4.4889999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.51000000000000001,"5min":0.49399999999999999,"15min":0.25},"max":{"1min":0.72599999999999998,"5min":0.84799999999999998,"15min":2.1349999999999998},"last":0.68799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738927,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175724,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196116,"statfs":{"total":96636764160,"available":96456822784,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.56100000000000005},"min":{"1min":0.44700000000000001,"5min":0.41699999999999998,"15min":0.192},"max":{"1min":0.67600000000000005,"5min":0.80900000000000005,"15min":5.5090000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.218},"max":{"1min":0.69699999999999995,"5min":0.95599999999999996,"15min":1.738},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58999999999999997,"15min":0.61299999999999999},"min":{"1min":0.499,"5min":0.45900000000000002,"15min":0.21199999999999999},"max":{"1min":0.71899999999999997,"5min":0.76000000000000001,"15min":5.5890000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.52900000000000003,"5min":0.48999999999999999,"15min":0.25800000000000001},"max":{"1min":0.753,"5min":1.5980000000000001,"15min":6.335},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:14:59.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-24T09:14:59.787 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T09:14:59.788 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-24T09:14:59.804 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":1851,"stamp":"2023-11-24T09:14:59.585696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":493240,"kb_used_data":12560,"kb_used_omap":4,"kb_used_meta":480635,"kb_avail":282622280,"statfs":{"total":289910292480,"available":289405214720,"internally_reserved":0,"allocated":12861440,"data_stored":7438583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":492170575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003519"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:23.988167+0000","last_change":"2023-11-24T08:29:04.820862+0000","last_active":"2023-11-24T09:12:23.988167+0000","last_peered":"2023-11-24T09:12:23.988167+0000","last_clean":"2023-11-24T09:12:23.988167+0000","last_became_active":"2023-11-24T08:29:04.818386+0000","last_became_peered":"2023-11-24T08:29:04.818386+0000","last_unstale":"2023-11-24T09:12:23.988167+0000","last_undegraded":"2023-11-24T09:12:23.988167+0000","last_fullsized":"2023-11-24T09:12:23.988167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:44:06.511981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:23.988123+0000","last_change":"2023-11-24T08:29:04.818405+0000","last_active":"2023-11-24T09:12:23.988123+0000","last_peered":"2023-11-24T09:12:23.988123+0000","last_clean":"2023-11-24T09:12:23.988123+0000","last_became_active":"2023-11-24T08:29:04.817967+0000","last_became_peered":"2023-11-24T08:29:04.817967+0000","last_unstale":"2023-11-24T09:12:23.988123+0000","last_undegraded":"2023-11-24T09:12:23.988123+0000","last_fullsized":"2023-11-24T09:12:23.988123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:51.528140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.143584+0000","last_change":"2023-11-24T08:29:04.819596+0000","last_active":"2023-11-24T09:11:10.143584+0000","last_peered":"2023-11-24T09:11:10.143584+0000","last_clean":"2023-11-24T09:11:10.143584+0000","last_became_active":"2023-11-24T08:29:04.818295+0000","last_became_peered":"2023-11-24T08:29:04.818295+0000","last_unstale":"2023-11-24T09:11:10.143584+0000","last_undegraded":"2023-11-24T09:11:10.143584+0000","last_fullsized":"2023-11-24T09:11:10.143584+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:22:22.642040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:53.992625+0000","last_change":"2023-11-24T08:29:04.820593+0000","last_active":"2023-11-24T09:12:53.992625+0000","last_peered":"2023-11-24T09:12:53.992625+0000","last_clean":"2023-11-24T09:12:53.992625+0000","last_became_active":"2023-11-24T08:29:04.818750+0000","last_became_peered":"2023-11-24T08:29:04.818750+0000","last_unstale":"2023-11-24T09:12:53.992625+0000","last_undegraded":"2023-11-24T09:12:53.992625+0000","last_fullsized":"2023-11-24T09:12:53.992625+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:17:33.495447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:49.272528+0000","last_change":"2023-11-24T08:29:04.820326+0000","last_active":"2023-11-24T09:12:49.272528+0000","last_peered":"2023-11-24T09:12:49.272528+0000","last_clean":"2023-11-24T09:12:49.272528+0000","last_became_active":"2023-11-24T08:29:04.819285+0000","last_became_peered":"2023-11-24T08:29:04.819285+0000","last_unstale":"2023-11-24T09:12:49.272528+0000","last_undegraded":"2023-11-24T09:12:49.272528+0000","last_fullsized":"2023-11-24T09:12:49.272528+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:42:03.353215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":492,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:48.885181+0000","last_change":"2023-11-24T08:29:04.819744+0000","last_active":"2023-11-24T09:12:48.885181+0000","last_peered":"2023-11-24T09:12:48.885181+0000","last_clean":"2023-11-24T09:12:48.885181+0000","last_became_active":"2023-11-24T08:29:04.819191+0000","last_became_peered":"2023-11-24T08:29:04.819191+0000","last_unstale":"2023-11-24T09:12:48.885181+0000","last_undegraded":"2023-11-24T09:12:48.885181+0000","last_fullsized":"2023-11-24T09:12:48.885181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:09:02.746576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":492,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:18.872164+0000","last_change":"2023-11-24T08:29:04.821790+0000","last_active":"2023-11-24T09:11:18.872164+0000","last_peered":"2023-11-24T09:11:18.872164+0000","last_clean":"2023-11-24T09:11:18.872164+0000","last_became_active":"2023-11-24T08:29:04.821369+0000","last_became_peered":"2023-11-24T08:29:04.821369+0000","last_unstale":"2023-11-24T09:11:18.872164+0000","last_undegraded":"2023-11-24T09:11:18.872164+0000","last_fullsized":"2023-11-24T09:11:18.872164+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:52:15.752249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:53.992589+0000","last_change":"2023-11-24T08:29:04.820777+0000","last_active":"2023-11-24T09:12:53.992589+0000","last_peered":"2023-11-24T09:12:53.992589+0000","last_clean":"2023-11-24T09:12:53.992589+0000","last_became_active":"2023-11-24T08:29:04.817968+0000","last_became_peered":"2023-11-24T08:29:04.817968+0000","last_unstale":"2023-11-24T09:12:53.992589+0000","last_undegraded":"2023-11-24T09:12:53.992589+0000","last_fullsized":"2023-11-24T09:12:53.992589+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:03:59.754900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":621,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.143030+0000","last_change":"2023-11-24T08:28:59.784048+0000","last_active":"2023-11-24T09:11:10.143030+0000","last_peered":"2023-11-24T09:11:10.143030+0000","last_clean":"2023-11-24T09:11:10.143030+0000","last_became_active":"2023-11-24T08:28:59.783498+0000","last_became_peered":"2023-11-24T08:28:59.783498+0000","last_unstale":"2023-11-24T09:11:10.143030+0000","last_undegraded":"2023-11-24T09:11:10.143030+0000","last_fullsized":"2023-11-24T09:11:10.143030+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_clean_scrub_stamp":"2023-11-24T08:28:58.760510+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:12:39.581115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738925,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165412,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206428,"statfs":{"total":96636764160,"available":96467382272,"internally_reserved":0,"allocated":3891200,"data_stored":2085949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:14:16 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.77300000000000002,"15min":0.625},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.17399999999999999},"max":{"1min":0.69599999999999995,"5min":25.623000000000001,"15min":25.623000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.90700000000000003,"15min":0.755},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.246},"max":{"1min":0.77600000000000002,"5min":25.969999999999999,"15min":25.969999999999999},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:14:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54800000000000004,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.20399999999999999},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":4.7060000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.51300000000000001,"5min":0.50700000000000001,"15min":0.23599999999999999},"max":{"1min":0.69699999999999995,"5min":4.2270000000000003,"15min":4.7549999999999999},"last":0.61299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738925,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152104,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94219736,"statfs":{"total":96636764160,"available":96481009664,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52900000000000003,"15min":0.53800000000000003},"min":{"1min":0.46200000000000002,"5min":0.432,"15min":0.17699999999999999},"max":{"1min":0.61599999999999999,"5min":0.69099999999999995,"15min":4.2889999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.53200000000000003,"5min":0.41199999999999998,"15min":0.20200000000000001},"max":{"1min":0.69099999999999995,"5min":0.78900000000000003,"15min":4.4130000000000003},"last":0.65800000000000003}]},{"osd":2,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.19800000000000001},"max":{"1min":0.72599999999999998,"5min":0.92600000000000005,"15min":4.4889999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.51000000000000001,"5min":0.49399999999999999,"15min":0.25},"max":{"1min":0.72599999999999998,"5min":0.84799999999999998,"15min":2.1349999999999998},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738928,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175724,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196116,"statfs":{"total":96636764160,"available":96456822784,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.56100000000000005},"min":{"1min":0.44700000000000001,"5min":0.41699999999999998,"15min":0.192},"max":{"1min":0.67600000000000005,"5min":0.80900000000000005,"15min":5.5090000000000003},"last":0.501},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.218},"max":{"1min":0.69699999999999995,"5min":0.95599999999999996,"15min":1.738},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58999999999999997,"15min":0.61299999999999999},"min":{"1min":0.499,"5min":0.45900000000000002,"15min":0.21199999999999999},"max":{"1min":0.71899999999999997,"5min":0.76000000000000001,"15min":5.5890000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.52900000000000003,"5min":0.48999999999999999,"15min":0.25800000000000001},"max":{"1min":0.753,"5min":1.5980000000000001,"15min":6.335},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:14:59.805 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-11-24T09:14:59.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-24T09:15:00.081 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T09:15:00.081 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-24T09:15:00.097 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":1851,"stamp":"2023-11-24T09:14:59.585696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":493240,"kb_used_data":12560,"kb_used_omap":4,"kb_used_meta":480635,"kb_avail":282622280,"statfs":{"total":289910292480,"available":289405214720,"internally_reserved":0,"allocated":12861440,"data_stored":7438583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":492170575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003519"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:23.988167+0000","last_change":"2023-11-24T08:29:04.820862+0000","last_active":"2023-11-24T09:12:23.988167+0000","last_peered":"2023-11-24T09:12:23.988167+0000","last_clean":"2023-11-24T09:12:23.988167+0000","last_became_active":"2023-11-24T08:29:04.818386+0000","last_became_peered":"2023-11-24T08:29:04.818386+0000","last_unstale":"2023-11-24T09:12:23.988167+0000","last_undegraded":"2023-11-24T09:12:23.988167+0000","last_fullsized":"2023-11-24T09:12:23.988167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:44:06.511981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:23.988123+0000","last_change":"2023-11-24T08:29:04.818405+0000","last_active":"2023-11-24T09:12:23.988123+0000","last_peered":"2023-11-24T09:12:23.988123+0000","last_clean":"2023-11-24T09:12:23.988123+0000","last_became_active":"2023-11-24T08:29:04.817967+0000","last_became_peered":"2023-11-24T08:29:04.817967+0000","last_unstale":"2023-11-24T09:12:23.988123+0000","last_undegraded":"2023-11-24T09:12:23.988123+0000","last_fullsized":"2023-11-24T09:12:23.988123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:51.528140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.143584+0000","last_change":"2023-11-24T08:29:04.819596+0000","last_active":"2023-11-24T09:11:10.143584+0000","last_peered":"2023-11-24T09:11:10.143584+0000","last_clean":"2023-11-24T09:11:10.143584+0000","last_became_active":"2023-11-24T08:29:04.818295+0000","last_became_peered":"2023-11-24T08:29:04.818295+0000","last_unstale":"2023-11-24T09:11:10.143584+0000","last_undegraded":"2023-11-24T09:11:10.143584+0000","last_fullsized":"2023-11-24T09:11:10.143584+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:22:22.642040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:53.992625+0000","last_change":"2023-11-24T08:29:04.820593+0000","last_active":"2023-11-24T09:12:53.992625+0000","last_peered":"2023-11-24T09:12:53.992625+0000","last_clean":"2023-11-24T09:12:53.992625+0000","last_became_active":"2023-11-24T08:29:04.818750+0000","last_became_peered":"2023-11-24T08:29:04.818750+0000","last_unstale":"2023-11-24T09:12:53.992625+0000","last_undegraded":"2023-11-24T09:12:53.992625+0000","last_fullsized":"2023-11-24T09:12:53.992625+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:17:33.495447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:49.272528+0000","last_change":"2023-11-24T08:29:04.820326+0000","last_active":"2023-11-24T09:12:49.272528+0000","last_peered":"2023-11-24T09:12:49.272528+0000","last_clean":"2023-11-24T09:12:49.272528+0000","last_became_active":"2023-11-24T08:29:04.819285+0000","last_became_peered":"2023-11-24T08:29:04.819285+0000","last_unstale":"2023-11-24T09:12:49.272528+0000","last_undegraded":"2023-11-24T09:12:49.272528+0000","last_fullsized":"2023-11-24T09:12:49.272528+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:42:03.353215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":492,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:48.885181+0000","last_change":"2023-11-24T08:29:04.819744+0000","last_active":"2023-11-24T09:12:48.885181+0000","last_peered":"2023-11-24T09:12:48.885181+0000","last_clean":"2023-11-24T09:12:48.885181+0000","last_became_active":"2023-11-24T08:29:04.819191+0000","last_became_peered":"2023-11-24T08:29:04.819191+0000","last_unstale":"2023-11-24T09:12:48.885181+0000","last_undegraded":"2023-11-24T09:12:48.885181+0000","last_fullsized":"2023-11-24T09:12:48.885181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:09:02.746576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":492,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:18.872164+0000","last_change":"2023-11-24T08:29:04.821790+0000","last_active":"2023-11-24T09:11:18.872164+0000","last_peered":"2023-11-24T09:11:18.872164+0000","last_clean":"2023-11-24T09:11:18.872164+0000","last_became_active":"2023-11-24T08:29:04.821369+0000","last_became_peered":"2023-11-24T08:29:04.821369+0000","last_unstale":"2023-11-24T09:11:18.872164+0000","last_undegraded":"2023-11-24T09:11:18.872164+0000","last_fullsized":"2023-11-24T09:11:18.872164+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:52:15.752249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:53.992589+0000","last_change":"2023-11-24T08:29:04.820777+0000","last_active":"2023-11-24T09:12:53.992589+0000","last_peered":"2023-11-24T09:12:53.992589+0000","last_clean":"2023-11-24T09:12:53.992589+0000","last_became_active":"2023-11-24T08:29:04.817968+0000","last_became_peered":"2023-11-24T08:29:04.817968+0000","last_unstale":"2023-11-24T09:12:53.992589+0000","last_undegraded":"2023-11-24T09:12:53.992589+0000","last_fullsized":"2023-11-24T09:12:53.992589+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:03:59.754900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":621,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.143030+0000","last_change":"2023-11-24T08:28:59.784048+0000","last_active":"2023-11-24T09:11:10.143030+0000","last_peered":"2023-11-24T09:11:10.143030+0000","last_clean":"2023-11-24T09:11:10.143030+0000","last_became_active":"2023-11-24T08:28:59.783498+0000","last_became_peered":"2023-11-24T08:28:59.783498+0000","last_unstale":"2023-11-24T09:11:10.143030+0000","last_undegraded":"2023-11-24T09:11:10.143030+0000","last_fullsized":"2023-11-24T09:11:10.143030+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_clean_scrub_stamp":"2023-11-24T08:28:58.760510+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:12:39.581115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738925,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165412,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206428,"statfs":{"total":96636764160,"available":96467382272,"internally_reserved":0,"allocated":3891200,"data_stored":2085949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:14:16 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.77300000000000002,"15min":0.625},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.17399999999999999},"max":{"1min":0.69599999999999995,"5min":25.623000000000001,"15min":25.623000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.90700000000000003,"15min":0.755},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.246},"max":{"1min":0.77600000000000002,"5min":25.969999999999999,"15min":25.969999999999999},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:14:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54800000000000004,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.20399999999999999},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":4.7060000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.51300000000000001,"5min":0.50700000000000001,"15min":0.23599999999999999},"max":{"1min":0.69699999999999995,"5min":4.2270000000000003,"15min":4.7549999999999999},"last":0.61299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738925,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152104,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94219736,"statfs":{"total":96636764160,"available":96481009664,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52900000000000003,"15min":0.53800000000000003},"min":{"1min":0.46200000000000002,"5min":0.432,"15min":0.17699999999999999},"max":{"1min":0.61599999999999999,"5min":0.69099999999999995,"15min":4.2889999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.53200000000000003,"5min":0.41199999999999998,"15min":0.20200000000000001},"max":{"1min":0.69099999999999995,"5min":0.78900000000000003,"15min":4.4130000000000003},"last":0.65800000000000003}]},{"osd":2,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.19800000000000001},"max":{"1min":0.72599999999999998,"5min":0.92600000000000005,"15min":4.4889999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.51000000000000001,"5min":0.49399999999999999,"15min":0.25},"max":{"1min":0.72599999999999998,"5min":0.84799999999999998,"15min":2.1349999999999998},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738928,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175724,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196116,"statfs":{"total":96636764160,"available":96456822784,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.56100000000000005},"min":{"1min":0.44700000000000001,"5min":0.41699999999999998,"15min":0.192},"max":{"1min":0.67600000000000005,"5min":0.80900000000000005,"15min":5.5090000000000003},"last":0.501},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.218},"max":{"1min":0.69699999999999995,"5min":0.95599999999999996,"15min":1.738},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58999999999999997,"15min":0.61299999999999999},"min":{"1min":0.499,"5min":0.45900000000000002,"15min":0.21199999999999999},"max":{"1min":0.71899999999999997,"5min":0.76000000000000001,"15min":5.5890000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.52900000000000003,"5min":0.48999999999999999,"15min":0.25800000000000001},"max":{"1min":0.753,"5min":1.5980000000000001,"15min":6.335},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:15:00.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-11-24T09:15:00.381 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T09:15:00.381 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":473,"fsid":"94e5c26f-c198-4207-b2c2-8bf1e80f7c68","created":"2023-11-24T08:28:47.691592+0000","modified":"2023-11-24T09:11:09.236805+0000","last_up_change":"2023-11-24T08:28:57.738525+0000","last_in_change":"2023-11-24T08:28:48.625743+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T08:28:58.672360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-24T08:29:03.081170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"36b343bc-77eb-4879-93e5-2965f034bc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6801","nonce":3990165559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6805","nonce":3990165559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6815","nonce":3990165559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":3990165559},{"type":"v1","addr":"172.21.15.153:6810","nonce":3990165559}]},"public_addr":"172.21.15.153:6801/3990165559","cluster_addr":"172.21.15.153:6805/3990165559","heartbeat_back_addr":"172.21.15.153:6815/3990165559","heartbeat_front_addr":"172.21.15.153:6810/3990165559","state":["exists","up"]},{"osd":1,"uuid":"f5d4f77c-1f30-4418-aa0a-2760e5b4d966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6804","nonce":822903425}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6808","nonce":822903425}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6813","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6814","nonce":822903425}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":822903425},{"type":"v1","addr":"172.21.15.153:6811","nonce":822903425}]},"public_addr":"172.21.15.153:6804/822903425","cluster_addr":"172.21.15.153:6808/822903425","heartbeat_back_addr":"172.21.15.153:6814/822903425","heartbeat_front_addr":"172.21.15.153:6811/822903425","state":["exists","up"]},{"osd":2,"uuid":"c0c18802-9220-4e2a-9ff6-81dc37fcdef8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6817","nonce":1195648817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6819","nonce":1195648817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6823","nonce":1195648817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1195648817},{"type":"v1","addr":"172.21.15.153:6821","nonce":1195648817}]},"public_addr":"172.21.15.153:6817/1195648817","cluster_addr":"172.21.15.153:6819/1195648817","heartbeat_back_addr":"172.21.15.153:6823/1195648817","heartbeat_front_addr":"172.21.15.153:6821/1195648817","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.207630+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.127077+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T08:28:54.177065+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":390,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T09:15:01.399 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2023-11-24T09:15:01.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-24T09:15:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-24T09:15:01.555 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T09:15:01.555 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-24T09:15:01.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2023-11-24T09:15:01.849 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 0 to deep-scrub 2023-11-24T09:15:01.867 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2023-11-24T09:15:01.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-24T09:15:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-24T09:15:02.010 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T09:15:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-24T09:15:02.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2023-11-24T09:15:02.280 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 1 to deep-scrub 2023-11-24T09:15:02.297 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2023-11-24T09:15:02.297 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-24T09:15:02.446 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-24T09:15:02.446 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T09:15:02.447 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-24T09:15:02.462 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2023-11-24T09:15:02.731 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 2 to deep-scrub 2023-11-24T09:15:02.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-24T09:15:03.006 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T09:15:03.006 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-24T09:15:03.021 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":1852,"stamp":"2023-11-24T09:15:01.586210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":493240,"kb_used_data":12560,"kb_used_omap":4,"kb_used_meta":480635,"kb_avail":282622280,"statfs":{"total":289910292480,"available":289405214720,"internally_reserved":0,"allocated":12861440,"data_stored":7438583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":492170575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:23.988167+0000","last_change":"2023-11-24T08:29:04.820862+0000","last_active":"2023-11-24T09:12:23.988167+0000","last_peered":"2023-11-24T09:12:23.988167+0000","last_clean":"2023-11-24T09:12:23.988167+0000","last_became_active":"2023-11-24T08:29:04.818386+0000","last_became_peered":"2023-11-24T08:29:04.818386+0000","last_unstale":"2023-11-24T09:12:23.988167+0000","last_undegraded":"2023-11-24T09:12:23.988167+0000","last_fullsized":"2023-11-24T09:12:23.988167+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:44:06.511981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:23.988123+0000","last_change":"2023-11-24T08:29:04.818405+0000","last_active":"2023-11-24T09:12:23.988123+0000","last_peered":"2023-11-24T09:12:23.988123+0000","last_clean":"2023-11-24T09:12:23.988123+0000","last_became_active":"2023-11-24T08:29:04.817967+0000","last_became_peered":"2023-11-24T08:29:04.817967+0000","last_unstale":"2023-11-24T09:12:23.988123+0000","last_undegraded":"2023-11-24T09:12:23.988123+0000","last_fullsized":"2023-11-24T09:12:23.988123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:51.528140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.143584+0000","last_change":"2023-11-24T08:29:04.819596+0000","last_active":"2023-11-24T09:11:10.143584+0000","last_peered":"2023-11-24T09:11:10.143584+0000","last_clean":"2023-11-24T09:11:10.143584+0000","last_became_active":"2023-11-24T08:29:04.818295+0000","last_became_peered":"2023-11-24T08:29:04.818295+0000","last_unstale":"2023-11-24T09:11:10.143584+0000","last_undegraded":"2023-11-24T09:11:10.143584+0000","last_fullsized":"2023-11-24T09:11:10.143584+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:22:22.642040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:53.992625+0000","last_change":"2023-11-24T08:29:04.820593+0000","last_active":"2023-11-24T09:12:53.992625+0000","last_peered":"2023-11-24T09:12:53.992625+0000","last_clean":"2023-11-24T09:12:53.992625+0000","last_became_active":"2023-11-24T08:29:04.818750+0000","last_became_peered":"2023-11-24T08:29:04.818750+0000","last_unstale":"2023-11-24T09:12:53.992625+0000","last_undegraded":"2023-11-24T09:12:53.992625+0000","last_fullsized":"2023-11-24T09:12:53.992625+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:17:33.495447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:49.272528+0000","last_change":"2023-11-24T08:29:04.820326+0000","last_active":"2023-11-24T09:12:49.272528+0000","last_peered":"2023-11-24T09:12:49.272528+0000","last_clean":"2023-11-24T09:12:49.272528+0000","last_became_active":"2023-11-24T08:29:04.819285+0000","last_became_peered":"2023-11-24T08:29:04.819285+0000","last_unstale":"2023-11-24T09:12:49.272528+0000","last_undegraded":"2023-11-24T09:12:49.272528+0000","last_fullsized":"2023-11-24T09:12:49.272528+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:42:03.353215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":492,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:48.885181+0000","last_change":"2023-11-24T08:29:04.819744+0000","last_active":"2023-11-24T09:12:48.885181+0000","last_peered":"2023-11-24T09:12:48.885181+0000","last_clean":"2023-11-24T09:12:48.885181+0000","last_became_active":"2023-11-24T08:29:04.819191+0000","last_became_peered":"2023-11-24T08:29:04.819191+0000","last_unstale":"2023-11-24T09:12:48.885181+0000","last_undegraded":"2023-11-24T09:12:48.885181+0000","last_fullsized":"2023-11-24T09:12:48.885181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:09:02.746576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":492,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:18.872164+0000","last_change":"2023-11-24T08:29:04.821790+0000","last_active":"2023-11-24T09:11:18.872164+0000","last_peered":"2023-11-24T09:11:18.872164+0000","last_clean":"2023-11-24T09:11:18.872164+0000","last_became_active":"2023-11-24T08:29:04.821369+0000","last_became_peered":"2023-11-24T08:29:04.821369+0000","last_unstale":"2023-11-24T09:11:18.872164+0000","last_undegraded":"2023-11-24T09:11:18.872164+0000","last_fullsized":"2023-11-24T09:11:18.872164+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:52:15.752249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":491,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:12:53.992589+0000","last_change":"2023-11-24T08:29:04.820777+0000","last_active":"2023-11-24T09:12:53.992589+0000","last_peered":"2023-11-24T09:12:53.992589+0000","last_clean":"2023-11-24T09:12:53.992589+0000","last_became_active":"2023-11-24T08:29:04.817968+0000","last_became_peered":"2023-11-24T08:29:04.817968+0000","last_unstale":"2023-11-24T09:12:53.992589+0000","last_undegraded":"2023-11-24T09:12:53.992589+0000","last_fullsized":"2023-11-24T09:12:53.992589+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:29:03.792449+0000","last_clean_scrub_stamp":"2023-11-24T08:29:03.792449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:03:59.754900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":621,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:11:10.143030+0000","last_change":"2023-11-24T08:28:59.784048+0000","last_active":"2023-11-24T09:11:10.143030+0000","last_peered":"2023-11-24T09:11:10.143030+0000","last_clean":"2023-11-24T09:11:10.143030+0000","last_became_active":"2023-11-24T08:28:59.783498+0000","last_became_peered":"2023-11-24T08:28:59.783498+0000","last_unstale":"2023-11-24T09:11:10.143030+0000","last_undegraded":"2023-11-24T09:11:10.143030+0000","last_fullsized":"2023-11-24T09:11:10.143030+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T08:28:58.760510+0000","last_clean_scrub_stamp":"2023-11-24T08:28:58.760510+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:12:39.581115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738925,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165412,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206428,"statfs":{"total":96636764160,"available":96467382272,"internally_reserved":0,"allocated":3891200,"data_stored":2085949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:14:16 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.77300000000000002,"15min":0.625},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.17399999999999999},"max":{"1min":0.69599999999999995,"5min":25.623000000000001,"15min":25.623000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.90700000000000003,"15min":0.755},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.246},"max":{"1min":0.77600000000000002,"5min":25.969999999999999,"15min":25.969999999999999},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Nov 24 09:14:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54800000000000004,"15min":0.56999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.20399999999999999},"max":{"1min":0.68400000000000005,"5min":0.80000000000000004,"15min":4.7060000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65600000000000003,"15min":0.64000000000000001},"min":{"1min":0.51300000000000001,"5min":0.50700000000000001,"15min":0.23599999999999999},"max":{"1min":0.69699999999999995,"5min":4.2270000000000003,"15min":4.7549999999999999},"last":0.61299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738925,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152104,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94219736,"statfs":{"total":96636764160,"available":96481009664,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52900000000000003,"15min":0.53800000000000003},"min":{"1min":0.46200000000000002,"5min":0.432,"15min":0.17699999999999999},"max":{"1min":0.61599999999999999,"5min":0.69099999999999995,"15min":4.2889999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.53200000000000003,"5min":0.41199999999999998,"15min":0.20200000000000001},"max":{"1min":0.69099999999999995,"5min":0.78900000000000003,"15min":4.4130000000000003},"last":0.65800000000000003}]},{"osd":2,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.19800000000000001},"max":{"1min":0.72599999999999998,"5min":0.92600000000000005,"15min":4.4889999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.51000000000000001,"5min":0.49399999999999999,"15min":0.25},"max":{"1min":0.72599999999999998,"5min":0.84799999999999998,"15min":2.1349999999999998},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738928,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175724,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196116,"statfs":{"total":96636764160,"available":96456822784,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.56100000000000005},"min":{"1min":0.44700000000000001,"5min":0.41699999999999998,"15min":0.192},"max":{"1min":0.67600000000000005,"5min":0.80900000000000005,"15min":5.5090000000000003},"last":0.501},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.218},"max":{"1min":0.69699999999999995,"5min":0.95599999999999996,"15min":1.738},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58999999999999997,"15min":0.61299999999999999},"min":{"1min":0.499,"5min":0.45900000000000002,"15min":0.21199999999999999},"max":{"1min":0.71899999999999997,"5min":0.76000000000000001,"15min":5.5890000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.52900000000000003,"5min":0.48999999999999999,"15min":0.25800000000000001},"max":{"1min":0.753,"5min":1.5980000000000001,"15min":6.335},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:15:03.025 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-11-24T08:29:03.792449+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=29, tm_sec=3, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=9, tm_min=15, tm_sec=0, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T09:15:03.025 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-11-24T08:29:03.792449+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=29, tm_sec=3, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=9, tm_min=15, tm_sec=0, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T09:15:03.025 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-11-24T08:29:03.792449+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=29, tm_sec=3, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=9, tm_min=15, tm_sec=0, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T09:15:03.025 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-11-24T08:29:03.792449+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=29, tm_sec=3, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=9, tm_min=15, tm_sec=0, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T09:15:03.025 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-11-24T08:29:03.792449+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=29, tm_sec=3, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=9, tm_min=15, tm_sec=0, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T09:15:03.025 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-11-24T08:29:03.792449+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=29, tm_sec=3, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=9, tm_min=15, tm_sec=0, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T09:15:03.025 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-11-24T08:29:03.792449+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=29, tm_sec=3, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=9, tm_min=15, tm_sec=0, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T09:15:03.026 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-11-24T08:29:03.792449+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=29, tm_sec=3, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=9, tm_min=15, tm_sec=0, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T09:15:03.026 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-11-24T08:28:58.760510+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=8, tm_min=28, tm_sec=58, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=9, tm_min=15, tm_sec=0, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T09:15:03.026 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-24T09:15:23.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-24T09:15:23.324 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-24T09:15:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-24T09:15:23.342 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":1862,"stamp":"2023-11-24T09:15:21.591055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":493224,"kb_used_data":12560,"kb_used_omap":4,"kb_used_meta":480635,"kb_avail":282622296,"statfs":{"total":289910292480,"available":289405231104,"internally_reserved":0,"allocated":12861440,"data_stored":7438583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":492170575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003024"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":500,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:15:13.793082+0000","last_change":"2023-11-24T09:15:13.793107+0000","last_active":"2023-11-24T09:15:13.793082+0000","last_peered":"2023-11-24T09:15:13.793082+0000","last_clean":"2023-11-24T09:15:13.793082+0000","last_became_active":"2023-11-24T08:29:04.818386+0000","last_became_peered":"2023-11-24T08:29:04.818386+0000","last_unstale":"2023-11-24T09:15:13.793082+0000","last_undegraded":"2023-11-24T09:15:13.793082+0000","last_fullsized":"2023-11-24T09:15:13.793082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T09:15:13.792939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T09:15:13.792939+0000","last_clean_scrub_stamp":"2023-11-24T09:15:13.792939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T19:57:26.067701+0000","scrub_duration":0.22230689200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":499,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:15:12.611346+0000","last_change":"2023-11-24T09:15:12.611395+0000","last_active":"2023-11-24T09:15:12.611346+0000","last_peered":"2023-11-24T09:15:12.611346+0000","last_clean":"2023-11-24T09:15:12.611346+0000","last_became_active":"2023-11-24T08:29:04.817967+0000","last_became_peered":"2023-11-24T08:29:04.817967+0000","last_unstale":"2023-11-24T09:15:12.611346+0000","last_undegraded":"2023-11-24T09:15:12.611346+0000","last_fullsized":"2023-11-24T09:15:12.611346+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T09:15:12.611116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T09:15:12.611116+0000","last_clean_scrub_stamp":"2023-11-24T09:15:12.611116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:47:01.347411+0000","scrub_duration":0.073551252999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":500,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:15:09.693812+0000","last_change":"2023-11-24T09:15:09.693844+0000","last_active":"2023-11-24T09:15:09.693812+0000","last_peered":"2023-11-24T09:15:09.693812+0000","last_clean":"2023-11-24T09:15:09.693812+0000","last_became_active":"2023-11-24T08:29:04.818295+0000","last_became_peered":"2023-11-24T08:29:04.818295+0000","last_unstale":"2023-11-24T09:15:09.693812+0000","last_undegraded":"2023-11-24T09:15:09.693812+0000","last_fullsized":"2023-11-24T09:15:09.693812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T09:15:09.693686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T09:15:09.693686+0000","last_clean_scrub_stamp":"2023-11-24T09:15:09.693686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:22:15.947903+0000","scrub_duration":0.18352280400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":500,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:15:10.547407+0000","last_change":"2023-11-24T09:15:10.547455+0000","last_active":"2023-11-24T09:15:10.547407+0000","last_peered":"2023-11-24T09:15:10.547407+0000","last_clean":"2023-11-24T09:15:10.547407+0000","last_became_active":"2023-11-24T08:29:04.818750+0000","last_became_peered":"2023-11-24T08:29:04.818750+0000","last_unstale":"2023-11-24T09:15:10.547407+0000","last_undegraded":"2023-11-24T09:15:10.547407+0000","last_fullsized":"2023-11-24T09:15:10.547407+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T09:15:10.547185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T09:15:10.547185+0000","last_clean_scrub_stamp":"2023-11-24T09:15:10.547185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:03:52.925042+0000","scrub_duration":0.0038702329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":500,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:15:07.317817+0000","last_change":"2023-11-24T09:15:07.317847+0000","last_active":"2023-11-24T09:15:07.317817+0000","last_peered":"2023-11-24T09:15:07.317817+0000","last_clean":"2023-11-24T09:15:07.317817+0000","last_became_active":"2023-11-24T08:29:04.819285+0000","last_became_peered":"2023-11-24T08:29:04.819285+0000","last_unstale":"2023-11-24T09:15:07.317817+0000","last_undegraded":"2023-11-24T09:15:07.317817+0000","last_fullsized":"2023-11-24T09:15:07.317817+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T09:15:07.317675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T09:15:07.317675+0000","last_clean_scrub_stamp":"2023-11-24T09:15:07.317675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:57:44.132047+0000","scrub_duration":0.071165177999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":501,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:15:03.270555+0000","last_change":"2023-11-24T09:15:03.270576+0000","last_active":"2023-11-24T09:15:03.270555+0000","last_peered":"2023-11-24T09:15:03.270555+0000","last_clean":"2023-11-24T09:15:03.270555+0000","last_became_active":"2023-11-24T08:29:04.819191+0000","last_became_peered":"2023-11-24T08:29:04.819191+0000","last_unstale":"2023-11-24T09:15:03.270555+0000","last_undegraded":"2023-11-24T09:15:03.270555+0000","last_fullsized":"2023-11-24T09:15:03.270555+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T09:15:03.270453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T09:15:03.270453+0000","last_clean_scrub_stamp":"2023-11-24T09:15:03.270453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:28:14.828629+0000","scrub_duration":0.16994216000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":502,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:15:07.228552+0000","last_change":"2023-11-24T09:15:07.228552+0000","last_active":"2023-11-24T09:15:07.228552+0000","last_peered":"2023-11-24T09:15:07.228552+0000","last_clean":"2023-11-24T09:15:07.228552+0000","last_became_active":"2023-11-24T08:29:04.821369+0000","last_became_peered":"2023-11-24T08:29:04.821369+0000","last_unstale":"2023-11-24T09:15:07.228552+0000","last_undegraded":"2023-11-24T09:15:07.228552+0000","last_fullsized":"2023-11-24T09:15:07.228552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T09:15:07.228401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T09:15:07.228401+0000","last_clean_scrub_stamp":"2023-11-24T09:15:07.228401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:13:29.389899+0000","scrub_duration":0.15494971799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":500,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:15:06.757820+0000","last_change":"2023-11-24T09:15:06.757843+0000","last_active":"2023-11-24T09:15:06.757820+0000","last_peered":"2023-11-24T09:15:06.757820+0000","last_clean":"2023-11-24T09:15:06.757820+0000","last_became_active":"2023-11-24T08:29:04.817968+0000","last_became_peered":"2023-11-24T08:29:04.817968+0000","last_unstale":"2023-11-24T09:15:06.757820+0000","last_undegraded":"2023-11-24T09:15:06.757820+0000","last_fullsized":"2023-11-24T09:15:06.757820+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T09:15:06.757698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T09:15:06.757698+0000","last_clean_scrub_stamp":"2023-11-24T09:15:06.757698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:21:50.282796+0000","scrub_duration":0.211084243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"473'99","reported_seq":633,"reported_epoch":473,"state":"active+clean","last_fresh":"2023-11-24T09:15:02.572900+0000","last_change":"2023-11-24T09:15:02.572900+0000","last_active":"2023-11-24T09:15:02.572900+0000","last_peered":"2023-11-24T09:15:02.572900+0000","last_clean":"2023-11-24T09:15:02.572900+0000","last_became_active":"2023-11-24T08:28:59.783498+0000","last_became_peered":"2023-11-24T08:28:59.783498+0000","last_unstale":"2023-11-24T09:15:02.572900+0000","last_undegraded":"2023-11-24T09:15:02.572900+0000","last_fullsized":"2023-11-24T09:15:02.572900+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'99","last_scrub_stamp":"2023-11-24T09:15:02.572754+0000","last_deep_scrub":"473'99","last_deep_scrub_stamp":"2023-11-24T09:15:02.572754+0000","last_clean_scrub_stamp":"2023-11-24T09:15:02.572754+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T20:47:13.259720+0000","scrub_duration":0.0095364820000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738929,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165408,"kb_used_data":3800,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206432,"statfs":{"total":96636764160,"available":96467386368,"internally_reserved":0,"allocated":3891200,"data_stored":2085949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165476805},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:15:18 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.501,"15min":0.623},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.17399999999999999},"max":{"1min":0.624,"5min":0.69599999999999995,"15min":25.623000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63600000000000001,"15min":0.753},"min":{"1min":0.53900000000000003,"5min":0.47899999999999998,"15min":0.246},"max":{"1min":0.753,"5min":0.77600000000000002,"15min":25.969999999999999},"last":0.71599999999999997}]},{"osd":1,"last update":"Fri Nov 24 09:15:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.55700000000000005},"min":{"1min":0.45100000000000001,"5min":0.434,"15min":0.20399999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":4.7060000000000004},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60999999999999999,"15min":0.629},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.23599999999999999},"max":{"1min":0.67000000000000004,"5min":0.753,"15min":4.7549999999999999},"last":0.54800000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738929,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152100,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":94219740,"statfs":{"total":96636764160,"available":96481013760,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151255493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52900000000000003,"15min":0.53800000000000003},"min":{"1min":0.46200000000000002,"5min":0.432,"15min":0.17699999999999999},"max":{"1min":0.61599999999999999,"5min":0.69099999999999995,"15min":4.2889999999999997},"last":0.624},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.58799999999999997},"min":{"1min":0.53200000000000003,"5min":0.41199999999999998,"15min":0.20200000000000001},"max":{"1min":0.69099999999999995,"5min":0.78900000000000003,"15min":4.4130000000000003},"last":0.68000000000000005}]},{"osd":2,"last update":"Fri Nov 24 09:14:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.19800000000000001},"max":{"1min":0.72599999999999998,"5min":0.92600000000000005,"15min":4.4889999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63800000000000001,"15min":0.63200000000000001},"min":{"1min":0.51000000000000001,"5min":0.49399999999999999,"15min":0.25},"max":{"1min":0.72599999999999998,"5min":0.84799999999999998,"15min":2.1349999999999998},"last":0.57399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738932,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175716,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":94196124,"statfs":{"total":96636764160,"available":96456830976,"internally_reserved":0,"allocated":4485120,"data_stored":2676317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175438277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54100000000000004,"15min":0.56100000000000005},"min":{"1min":0.44700000000000001,"5min":0.41699999999999998,"15min":0.192},"max":{"1min":0.67600000000000005,"5min":0.80900000000000005,"15min":5.5090000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.218},"max":{"1min":0.69699999999999995,"5min":0.95599999999999996,"15min":1.738},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Nov 24 09:14:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58999999999999997,"15min":0.61299999999999999},"min":{"1min":0.499,"5min":0.45900000000000002,"15min":0.21199999999999999},"max":{"1min":0.71899999999999997,"5min":0.76000000000000001,"15min":5.5890000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.63700000000000001,"15min":0.63500000000000001},"min":{"1min":0.52900000000000003,"5min":0.48999999999999999,"15min":0.25800000000000001},"max":{"1min":0.753,"5min":1.5980000000000001,"15min":6.335},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T09:15:23.343 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2023-11-24T09:15:23.659 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2771, in wait_for_clean cur_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-24T09:15:23.660 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-24T09:15:23.660 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-24T09:15:23.660 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-11-24T09:15:23.660 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:15:23.751 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-11-24T09:15:23.751 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-11-24T09:15:23.751 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:15:27.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2023-11-24T09:15:29.853 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-11-24T09:15:29.854 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-11-24T09:15:29.854 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:15:29.954 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-11-24T09:15:29.955 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-24T09:15:29.955 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-11-24T09:15:29.955 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:15:29.986 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-11-24T09:15:29.986 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-24T09:15:29.986 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-11-24T09:15:29.986 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T09:15:30.062 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-11-24T09:15:30.062 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-24T09:15:30.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-11-24T09:15:30.118 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T09:15:30.118 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-11-24T09:15:30.374 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T09:15:30.375 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-11-24T09:15:30.540 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T09:15:30.541 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-11-24T09:15:30.673 INFO:tasks.ceph:Archiving mon data... 2023-11-24T09:15:30.673 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501/data/cluster1.mon.a.tgz 2023-11-24T09:15:30.673 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-11-24T09:15:30.678 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.TSXlsTLHwX 2023-11-24T09:15:30.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.TSXlsTLHwX 2023-11-24T09:15:30.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.TSXlsTLHwX 2023-11-24T09:15:30.865 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.TSXlsTLHwX is 335KB 2023-11-24T09:15:30.926 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.TSXlsTLHwX 2023-11-24T09:15:30.931 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-24T09:15:30.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-11-24T09:15:30.973 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-11-24T09:15:30.988 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-24T09:15:30.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-24T09:15:31.029 DEBUG:teuthology.orchestra.run.smithi164:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-24T09:15:31.053 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-24T09:15:31.109 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-24T09:15:31.113 INFO:tasks.ceph:Archiving crash dumps... 2023-11-24T09:15:31.115 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501/remote/smithi153/crash 2023-11-24T09:15:31.116 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-24T09:15:31.127 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501/remote/smithi164/crash 2023-11-24T09:15:31.128 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-24T09:15:31.183 INFO:tasks.ceph:Compressing logs... 2023-11-24T09:15:31.183 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-24T09:15:31.185 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-24T09:15:31.206 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34661.log 2023-11-24T09:15:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39330.log 2023-11-24T09:15:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.34661.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34661.log.gz 2023-11-24T09:15:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-11-24T09:15:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34685.log 2023-11-24T09:15:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39330.log.gz 2023-11-24T09:15:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-11-24T09:15:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.34685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39122.log 2023-11-24T09:15:31.209 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34685.log.gz 2023-11-24T09:15:31.209 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39181.log 2023-11-24T09:15:31.211 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34744.log 2023-11-24T09:15:31.211 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39122.log.gz 2023-11-24T09:15:31.211 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39181.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/cluster1-client.admin.39181.log.gz 2023-11-24T09:15:31.211 INFO:teuthology.orchestra.run.smithi153.stderr: /var/log/ceph/cluster1-client.admin.41219.log 2023-11-24T09:15:31.211 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.34744.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34744.log.gz 2023-11-24T09:15:31.211 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30936.log 2023-11-24T09:15:31.211 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.41219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41195.log 2023-11-24T09:15:31.212 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41219.log.gz 2023-11-24T09:15:31.212 INFO:teuthology.orchestra.run.smithi153.stderr: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-11-24T09:15:31.212 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.tmp-client.admin.30936.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30936.log.gz 2023-11-24T09:15:31.212 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-11-24T09:15:31.212 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.41195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39019.log 2023-11-24T09:15:31.212 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41195.log.gz 2023-11-24T09:15:31.214 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41143.log 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39182.log 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39019.log.gz 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.41143.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41143.log.gz 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39260.log 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39182.log.gz 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39260.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.39260.log.gz -5 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.41091.log 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41169.log 2023-11-24T09:15:31.215 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.41091.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41091.log.gz 2023-11-24T09:15:31.216 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-11-24T09:15:31.216 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.41169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34567.log 2023-11-24T09:15:31.216 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41169.log.gz 2023-11-24T09:15:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39177.log 2023-11-24T09:15:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.34567.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39460.log 2023-11-24T09:15:31.218 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.34567.log.gz 2023-11-24T09:15:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39434.log 2023-11-24T09:15:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39177.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39177.log.gz/var/log/ceph/cluster1-client.admin.39460.log: 2023-11-24T09:15:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40932.log 2023-11-24T09:15:31.219 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39460.log.gz 2023-11-24T09:15:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39434.log.gz 2023-11-24T09:15:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39070.log 2023-11-24T09:15:31.219 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.40932.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40932.log.gz -5 2023-11-24T09:15:31.219 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.39148.log 2023-11-24T09:15:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38958.log 2023-11-24T09:15:31.222 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39070.log.gz/var/log/ceph/cluster1-client.admin.39148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34812.log 2023-11-24T09:15:31.222 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-11-24T09:15:31.222 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39148.log.gz 2023-11-24T09:15:31.222 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.38958.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38958.log.gz 2023-11-24T09:15:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39408.log 2023-11-24T09:15:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.34812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39252.log 2023-11-24T09:15:31.223 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34812.log.gz 2023-11-24T09:15:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41062.log 2023-11-24T09:15:31.223 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39408.log.gz/var/log/ceph/cluster1-client.admin.39252.log: 2023-11-24T09:15:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41117.log 2023-11-24T09:15:31.223 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39252.log.gz 2023-11-24T09:15:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.41062.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41062.log.gz 2023-11-24T09:15:31.223 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39338.log 2023-11-24T09:15:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.41117.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40984.log 2023-11-24T09:15:31.225 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.41117.log.gz 2023-11-24T09:15:31.225 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39338.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-11-24T09:15:31.225 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39338.log.gz/var/log/ceph/cluster1-client.admin.40984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40958.log 2023-11-24T09:15:31.225 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-11-24T09:15:31.225 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40984.log.gz 2023-11-24T09:15:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41010.log 2023-11-24T09:15:31.228 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39096.log 2023-11-24T09:15:31.229 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.40958.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40958.log.gz 2023-11-24T09:15:31.229 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.41010.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41010.log.gz 2023-11-24T09:15:31.229 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34709.log 2023-11-24T09:15:31.230 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40906.log 2023-11-24T09:15:31.230 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39096.log: /var/log/ceph/cluster1-client.admin.34709.log: 87.9% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39096.log.gz 2023-11-24T09:15:31.230 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/cluster1.log.gz 2023-11-24T09:15:31.230 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34709.log.gz 2023-11-24T09:15:31.230 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39334.log 2023-11-24T09:15:31.232 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39258.log 2023-11-24T09:15:31.232 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.40906.log: /var/log/ceph/cluster1-client.admin.39334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41036.log 2023-11-24T09:15:31.232 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40906.log.gz 2023-11-24T09:15:31.232 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0%/var/log/ceph/cluster1-client.admin.39258.log: -- replaced with /var/log/ceph/cluster1-client.admin.39334.log.gz 2023-11-24T09:15:31.232 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.39258.log.gz 2023-11-24T09:15:31.233 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.39044.log 2023-11-24T09:15:31.233 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-11-24T09:15:31.233 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.41036.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41036.log.gz 2023-11-24T09:15:31.233 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-client.admin.39044.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39044.log.gz 2023-11-24T09:15:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57537.log 2023-11-24T09:15:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70360.log 2023-11-24T09:15:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67505.log 2023-11-24T09:15:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76080.log 2023-11-24T09:15:31.239 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70360.log.gz 2023-11-24T09:15:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67505.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57537.log.gz 2023-11-24T09:15:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76471.log 2023-11-24T09:15:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67968.log 2023-11-24T09:15:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76080.log: 84.0%/var/log/ceph/cluster1-client.mirror.76471.log: -- replaced with /var/log/ceph/cluster1-client.mirror.76080.log.gz 2023-11-24T09:15:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64111.log 2023-11-24T09:15:31.241 INFO:teuthology.orchestra.run.smithi164.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76471.log.gz 2023-11-24T09:15:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42237.log 2023-11-24T09:15:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67968.log: /var/log/ceph/cluster2-client.mirror.64111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52437.log 2023-11-24T09:15:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42237.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64111.log.gz 2023-11-24T09:15:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74308.log 2023-11-24T09:15:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50044.log 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52437.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67505.log.gz 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54831.log 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67968.log.gz 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50044.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65421.log 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50044.log.gz 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52437.log.gz 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74308.log.gz 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70630.log 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54787.log 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.53481.log 2023-11-24T09:15:31.244 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70630.log.gz 2023-11-24T09:15:31.245 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39926.log 2023-11-24T09:15:31.247 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54787.log.gz 92.4%gzip 2023-11-24T09:15:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.53481.log: -- replaced with /var/log/ceph/cluster2-client.mirror.54831.log.gz 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.79738.log 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.39926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65506.log 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65421.log.gz 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61708.log 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79738.log.gz 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71395.log 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42237.log.gz 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39926.log.gz 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71553.log 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61708.log.gz 2023-11-24T09:15:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71395.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65506.log.gz 2023-11-24T09:15:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.43343.log 2023-11-24T09:15:31.249 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71395.log.gz 2023-11-24T09:15:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64754.log 2023-11-24T09:15:31.249 INFO:teuthology.orchestra.run.smithi164.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71553.log.gz 2023-11-24T09:15:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.43343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64668.log 2023-11-24T09:15:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:gzip/var/log/ceph/cluster2-client.mirror.64754.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48324.log 2023-11-24T09:15:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51278.log 2023-11-24T09:15:31.255 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64668.log.gz 2023-11-24T09:15:31.255 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64754.log.gz 2023-11-24T09:15:31.255 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75115.log 2023-11-24T09:15:31.255 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54285.log 2023-11-24T09:15:31.255 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51278.log: /var/log/ceph/cluster2-client.mirror.75115.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48324.log.gz 2023-11-24T09:15:31.255 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66960.log 2023-11-24T09:15:31.255 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48497.log 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54285.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51278.log.gz 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.43343.log.gz 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57759.log 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54285.log.gz 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48497.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66960.log.gz 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68629.log 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48497.log.gz 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61823.log 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64576.log 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75115.log.gz 2023-11-24T09:15:31.256 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52279.log 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62348.log 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61823.log.gz 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64576.log.gz 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70473.log 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57759.log.gz 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52457.log 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79036.log 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52279.log.gz 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70473.log.gz 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68629.log.gz 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67290.log 2023-11-24T09:15:31.257 INFO:teuthology.orchestra.run.smithi164.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52457.log.gz 2023-11-24T09:15:31.258 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78828.log 2023-11-24T09:15:31.258 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79036.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79036.log.gz 2023-11-24T09:15:31.258 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68257.log 2023-11-24T09:15:31.258 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62348.log.gz 2023-11-24T09:15:31.258 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78828.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78828.log.gz 2023-11-24T09:15:31.258 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41812.log 2023-11-24T09:15:31.258 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58834.log 2023-11-24T09:15:31.258 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45391.log 2023-11-24T09:15:31.258 INFO:teuthology.orchestra.run.smithi164.stderr: 82.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41812.log.gz 2023-11-24T09:15:31.271 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48178.log 2023-11-24T09:15:31.271 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67290.log.gz 2023-11-24T09:15:31.271 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55597.log 2023-11-24T09:15:31.271 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58834.log.gz 2023-11-24T09:15:31.271 INFO:teuthology.orchestra.run.smithi164.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45391.log.gz 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74988.log 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48178.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48178.log.gz 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39256.log 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68257.log.gz 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63456.log 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55597.log.gz 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50123.log 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39256.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74988.log.gz 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39256.log.gz 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66302.log 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49546.log 2023-11-24T09:15:31.272 INFO:teuthology.orchestra.run.smithi164.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50123.log.gz 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48885.log 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49546.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63456.log.gz 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79192.log 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49546.log.gz 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68857.log 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48885.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66302.log.gz 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79192.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55655.log 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79192.log.gz 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68857.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54371.log 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48885.log.gz 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55655.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68857.log.gz 2023-11-24T09:15:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62564.log 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55655.log.gz 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56441.log 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60572.log 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56441.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54371.log.gz 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55465.log 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60572.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56441.log.gz 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46243.log 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55465.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62564.log.gz 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60572.log.gz 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77606.log 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55465.log.gz 2023-11-24T09:15:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73861.log 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77606.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69278.log 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77606.log.gz 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.73861.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66836.log 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69278.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73861.log.gz 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54447.log 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53644.log 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66836.log: /var/log/ceph/cluster2-client.mirror.54447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69898.log 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66836.log.gz 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54447.log.gz 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72342.log 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76656.log 2023-11-24T09:15:31.275 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69278.log.gz 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73988.log 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53644.log.gz 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72342.log.gz 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39066.log 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76656.log: /var/log/ceph/cluster2-client.mirror.73988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79426.log 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76656.log.gz 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42193.log 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39066.log.gz 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55183.log 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79426.log.gz 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46243.log.gz 2023-11-24T09:15:31.276 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69898.log.gz 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46552.log 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42193.log: /var/log/ceph/cluster2-client.mirror.55183.log: gzip 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42193.log.gz -5 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.71935.log 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55183.log.gz 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73988.log.gz 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63746.log 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39634.log 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71935.log.gz 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52400.log 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.39634.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63746.log.gz 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39634.log.gz 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70048.log 2023-11-24T09:15:31.277 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.53040.log 2023-11-24T09:15:31.278 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52400.log: /var/log/ceph/cluster1-client.mirror.70048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59546.log 2023-11-24T09:15:31.278 INFO:teuthology.orchestra.run.smithi164.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70048.log.gz 2023-11-24T09:15:31.278 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.53040.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53417.log 2023-11-24T09:15:31.278 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39759.log 2023-11-24T09:15:31.278 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59546.log.gz 2023-11-24T09:15:31.278 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53417.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50518.log 2023-11-24T09:15:31.278 INFO:teuthology.orchestra.run.smithi164.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53417.log.gz 2023-11-24T09:15:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5/var/log/ceph/cluster2-client.admin.39759.log: --verbose -- /var/log/ceph/cluster1-client.mirror.64374.log 2023-11-24T09:15:31.286 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39759.log.gz 2023-11-24T09:15:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57897.log 2023-11-24T09:15:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64374.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50518.log.gz 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59028.log 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57897.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64374.log.gz 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61401.log 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57897.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.46552.log.gz 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75580.log 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48534.log 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61401.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59028.log.gz 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71193.log 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75580.log.gz 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48534.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61401.log.gz 2023-11-24T09:15:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62644.log 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79348.log 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48534.log.gz 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71193.log.gz 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65377.log 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48927.log 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79348.log.gz 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65377.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62644.log.gz 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42471.log 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65377.log.gz 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69320.log 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48927.log: /var/log/ceph/cluster2-client.mirror.42471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34905.log 2023-11-24T09:15:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59792.log 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.34905.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69320.log.gz 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75053.log 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34905.log.gz 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42471.log.gz 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59792.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59792.log.gz 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44774.log 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68133.log 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75053.log.gz 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39015.log 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.45827.log 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44774.log.gz 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39015.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39015.log.gz -5 2023-11-24T09:15:31.289 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.46423.log 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.45827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59431.log 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65987.log 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59431.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68133.log.gz 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64157.log 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65987.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59431.log.gz 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65987.log.gz 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77361.log 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42912.log 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64157.log: /var/log/ceph/cluster1-client.admin.77361.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77361.log.gz 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64157.log.gz 2023-11-24T09:15:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55768.log 2023-11-24T09:15:31.291 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40368.log 2023-11-24T09:15:31.291 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42912.log: /var/log/ceph/cluster2-client.mirror.55768.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42912.log.gz 2023-11-24T09:15:31.291 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52737.log 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46423.log.gz 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.40368.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.40368.log.gz 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62820.log 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55768.log.gz 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62779.log 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52737.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69630.log 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62820.log: /var/log/ceph/cluster1-client.mirror.62779.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62779.log.gz 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62820.log.gz 2023-11-24T09:15:31.295 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57515.log 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68071.log 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57515.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69630.log.gz 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48714.log 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74908.log 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48714.log: /var/log/ceph/cluster2-client.mirror.68071.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48714.log.gz 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57515.log.gz 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68071.log.gz 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49235.log 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71013.log 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.49235.log: -- replaced with /var/log/ceph/cluster2-client.mirror.74908.log.gz 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53295.log 2023-11-24T09:15:31.296 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71013.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71013.log.gz 2023-11-24T09:15:31.299 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49235.log.gz 2023-11-24T09:15:31.299 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58029.log 2023-11-24T09:15:31.299 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44385.log 2023-11-24T09:15:31.299 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53295.log: /var/log/ceph/cluster2-client.mirror.58029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39821.log 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44385.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44385.log.gz 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.53040.log.gz 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58029.log.gz 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72138.log 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39821.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74560.log 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69154.log 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72138.log.gz 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74560.log: 86.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74560.log.gz 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61482.log 2023-11-24T09:15:31.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44121.log 2023-11-24T09:15:31.301 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61482.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61482.log.gz 2023-11-24T09:15:31.301 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75009.log 2023-11-24T09:15:31.302 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44121.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69154.log.gz 2023-11-24T09:15:31.302 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54741.log 2023-11-24T09:15:31.304 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75009.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75009.log.gz 2023-11-24T09:15:31.305 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56259.log 2023-11-24T09:15:31.305 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54741.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54741.log.gz 2023-11-24T09:15:31.306 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53273.log 2023-11-24T09:15:31.307 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56259.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56259.log.gz 2023-11-24T09:15:31.307 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44121.log.gz 2023-11-24T09:15:31.308 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60797.log 2023-11-24T09:15:31.308 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42216.log 2023-11-24T09:15:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60797.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53273.log.gz 2023-11-24T09:15:31.309 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60797.log.gz 2023-11-24T09:15:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66478.log 2023-11-24T09:15:31.310 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69340.log 2023-11-24T09:15:31.310 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42216.log.gz/var/log/ceph/cluster1-client.mirror.66478.log: 2023-11-24T09:15:31.310 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66010.log 2023-11-24T09:15:31.311 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69340.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66478.log.gz 2023-11-24T09:15:31.312 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71238.log 2023-11-24T09:15:31.312 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66010.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66010.log.gz 2023-11-24T09:15:31.313 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47288.log 2023-11-24T09:15:31.313 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69340.log.gz 2023-11-24T09:15:31.314 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46335.log 2023-11-24T09:15:31.314 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71238.log.gz 2023-11-24T09:15:31.314 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47288.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50684.log 2023-11-24T09:15:31.316 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46335.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46335.log.gz 2023-11-24T09:15:31.316 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57279.log 2023-11-24T09:15:31.317 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50684.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50684.log.gz 2023-11-24T09:15:31.318 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44078.log 2023-11-24T09:15:31.320 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57279.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47288.log.gz 2023-11-24T09:15:31.320 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57279.log.gz 2023-11-24T09:15:31.320 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55635.log 2023-11-24T09:15:31.321 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42514.log 2023-11-24T09:15:31.322 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55635.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55635.log.gz 2023-11-24T09:15:31.323 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76678.log 2023-11-24T09:15:31.323 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42514.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42514.log.gz 2023-11-24T09:15:31.324 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47973.log 2023-11-24T09:15:31.327 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76678.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44078.log.gz 2023-11-24T09:15:31.338 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.40679.log 2023-11-24T09:15:31.341 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47973.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47973.log.gz 2023-11-24T09:15:31.341 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54493.log 2023-11-24T09:15:31.345 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.40679.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.2.45827.log.gz 2023-11-24T09:15:31.351 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.53481.log.gz 2023-11-24T09:15:31.352 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54427.log 2023-11-24T09:15:31.352 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54493.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66214.log 2023-11-24T09:15:31.353 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54427.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54493.log.gz 2023-11-24T09:15:31.353 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78126.log 2023-11-24T09:15:31.353 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66214.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54427.log.gz 2023-11-24T09:15:31.354 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69609.log 2023-11-24T09:15:31.354 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78126.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78126.log.gz 2023-11-24T09:15:31.355 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53332.log 2023-11-24T09:15:31.355 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3%/var/log/ceph/cluster2-client.mirror.69609.log: -- replaced with /var/log/ceph/cluster2-client.mirror.66214.log.gz 2023-11-24T09:15:31.356 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57626.log 2023-11-24T09:15:31.356 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53332.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69609.log.gz 2023-11-24T09:15:31.356 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39176.log 2023-11-24T09:15:31.357 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57626.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53332.log.gz 2023-11-24T09:15:31.358 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45325.log 2023-11-24T09:15:31.358 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57626.log.gz 2023-11-24T09:15:31.358 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39176.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39176.log.gz 2023-11-24T09:15:31.358 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56975.log 2023-11-24T09:15:31.358 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50061.log 2023-11-24T09:15:31.360 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56975.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56975.log.gz 2023-11-24T09:15:31.360 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69382.log 2023-11-24T09:15:31.360 INFO:teuthology.orchestra.run.smithi164.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45325.log.gz 2023-11-24T09:15:31.360 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78516.log 2023-11-24T09:15:31.362 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69382.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50061.log.gz 2023-11-24T09:15:31.362 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69382.log.gz 2023-11-24T09:15:31.362 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67084.log 2023-11-24T09:15:31.362 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66639.log 2023-11-24T09:15:31.362 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78516.log.gz 2023-11-24T09:15:31.362 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65141.log 2023-11-24T09:15:31.365 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66639.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.40679.log.gz 2023-11-24T09:15:31.365 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67084.log.gz 2023-11-24T09:15:31.365 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66639.log.gz 2023-11-24T09:15:31.365 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73567.log 2023-11-24T09:15:31.365 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63392.log 2023-11-24T09:15:31.365 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73567.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65141.log.gz 2023-11-24T09:15:31.365 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65967.log 2023-11-24T09:15:31.365 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55555.log 2023-11-24T09:15:31.366 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3%/var/log/ceph/cluster1-client.mirror.63392.log: -- replaced with /var/log/ceph/cluster2-client.mirror.73567.log.gz 2023-11-24T09:15:31.366 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65967.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63392.log.gz 2023-11-24T09:15:31.366 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55747.log 2023-11-24T09:15:31.367 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55555.log: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78334.log 2023-11-24T09:15:31.367 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65967.log.gz 2023-11-24T09:15:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55747.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55747.log.gz 2023-11-24T09:15:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66814.log 2023-11-24T09:15:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78334.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59255.log 2023-11-24T09:15:31.368 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55555.log.gz 2023-11-24T09:15:31.368 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78334.log.gz 2023-11-24T09:15:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62954.log 2023-11-24T09:15:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66814.log: /var/log/ceph/cluster2-client.mirror.59255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58243.log 2023-11-24T09:15:31.369 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62954.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66814.log.gz 2023-11-24T09:15:31.369 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62954.log.gz 2023-11-24T09:15:31.370 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59255.log.gz 2023-11-24T09:15:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46724.log 2023-11-24T09:15:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49763.log 2023-11-24T09:15:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58243.log: /var/log/ceph/cluster2-client.mirror.46724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61998.log 2023-11-24T09:15:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49763.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58243.log.gz 2023-11-24T09:15:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63323.log 2023-11-24T09:15:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61998.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46724.log.gz 2023-11-24T09:15:31.376 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61998.log.gz 2023-11-24T09:15:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52121.log 2023-11-24T09:15:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.41812.log 2023-11-24T09:15:31.377 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63323.log: /var/log/ceph/cluster2-client.mirror.52121.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63323.log.gz 2023-11-24T09:15:31.377 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72228.log 2023-11-24T09:15:31.378 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52121.log.gz 2023-11-24T09:15:31.378 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.41812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68712.log 2023-11-24T09:15:31.379 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72228.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72228.log.gz 2023-11-24T09:15:31.379 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52913.log 2023-11-24T09:15:31.381 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68712.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68712.log.gz 2023-11-24T09:15:31.381 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70135.log 2023-11-24T09:15:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52913.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52913.log.gz 2023-11-24T09:15:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77079.log 2023-11-24T09:15:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70135.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70135.log.gz 2023-11-24T09:15:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66898.log 2023-11-24T09:15:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.77079.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.41812.log.gz 2023-11-24T09:15:31.385 INFO:teuthology.orchestra.run.smithi164.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.77079.log.gz 2023-11-24T09:15:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48756.log 2023-11-24T09:15:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49255.log 2023-11-24T09:15:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66898.log: /var/log/ceph/cluster2-client.mirror.48756.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48756.log.gz 2023-11-24T09:15:31.386 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66898.log.gz 2023-11-24T09:15:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39659.log 2023-11-24T09:15:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53253.log 2023-11-24T09:15:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49255.log: /var/log/ceph/cluster1-client.admin.39659.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39659.log.gz 2023-11-24T09:15:31.388 INFO:teuthology.orchestra.run.smithi164.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49255.log.gz 2023-11-24T09:15:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54216.log 2023-11-24T09:15:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76965.log 2023-11-24T09:15:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53253.log: /var/log/ceph/cluster2-client.mirror.54216.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54216.log.gz 2023-11-24T09:15:31.389 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53253.log.gz 2023-11-24T09:15:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51546.log 2023-11-24T09:15:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70788.log 2023-11-24T09:15:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76965.log: /var/log/ceph/cluster2-client.mirror.51546.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.76965.log.gz 2023-11-24T09:15:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62977.log 2023-11-24T09:15:31.391 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51546.log.gz 2023-11-24T09:15:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70788.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70788.log.gz 2023-11-24T09:15:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64267.log 2023-11-24T09:15:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71283.log 2023-11-24T09:15:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62977.log: /var/log/ceph/cluster1-client.mirror.64267.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62977.log.gz 2023-11-24T09:15:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48694.log 2023-11-24T09:15:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71283.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71283.log.gz 2023-11-24T09:15:31.394 INFO:teuthology.orchestra.run.smithi164.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64267.log.gz 2023-11-24T09:15:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75835.log 2023-11-24T09:15:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48694.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48694.log.gz 2023-11-24T09:15:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62888.log 2023-11-24T09:15:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50081.log 2023-11-24T09:15:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75835.log: /var/log/ceph/cluster2-client.mirror.62888.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75835.log.gz 2023-11-24T09:15:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50938.log 2023-11-24T09:15:31.397 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8%/var/log/ceph/cluster2-client.mirror.50081.log: -- replaced with /var/log/ceph/cluster2-client.mirror.62888.log.gz 2023-11-24T09:15:31.397 INFO:teuthology.orchestra.run.smithi164.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50081.log.gz 2023-11-24T09:15:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76235.log 2023-11-24T09:15:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64927.log 2023-11-24T09:15:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50938.log: /var/log/ceph/cluster2-client.mirror.76235.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76235.log.gz 2023-11-24T09:15:31.399 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50938.log.gz 2023-11-24T09:15:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42537.log 2023-11-24T09:15:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40453.log 2023-11-24T09:15:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64927.log: /var/log/ceph/cluster1-client.mirror.42537.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64927.log.gz 2023-11-24T09:15:31.401 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42537.log.gz 2023-11-24T09:15:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62172.log 2023-11-24T09:15:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61223.log 2023-11-24T09:15:31.402 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40453.log: /var/log/ceph/cluster2-client.mirror.62172.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40453.log.gz 2023-11-24T09:15:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75918.log 2023-11-24T09:15:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61223.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62172.log.gz 2023-11-24T09:15:31.403 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61223.log.gz 2023-11-24T09:15:31.404 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78542.log 2023-11-24T09:15:31.404 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56079.log 2023-11-24T09:15:31.404 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75918.log: /var/log/ceph/cluster2-client.admin.78542.log: 0.0% 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75918.log.gz 2023-11-24T09:15:31.404 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.78542.log.gz 2023-11-24T09:15:31.404 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56775.log 2023-11-24T09:15:31.405 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47993.log 2023-11-24T09:15:31.405 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56079.log: /var/log/ceph/cluster2-client.mirror.56775.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56079.log.gz 2023-11-24T09:15:31.405 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78022.log 2023-11-24T09:15:31.407 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47993.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47993.log.gz 2023-11-24T09:15:31.407 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56775.log.gz 2023-11-24T09:15:31.407 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45827.log 2023-11-24T09:15:31.407 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63114.log 2023-11-24T09:15:31.408 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78022.log: /var/log/ceph/cluster2-client.mirror.45827.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78022.log.gz 2023-11-24T09:15:31.408 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74494.log 2023-11-24T09:15:31.408 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63114.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63114.log.gz 2023-11-24T09:15:31.409 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68836.log 2023-11-24T09:15:31.409 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74494.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74494.log.gz 2023-11-24T09:15:31.410 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45198.log 2023-11-24T09:15:31.411 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68836.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68836.log.gz 2023-11-24T09:15:31.411 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46873.log 2023-11-24T09:15:31.415 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45198.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45827.log.gz 2023-11-24T09:15:31.422 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67270.log 2023-11-24T09:15:31.428 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46873.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46873.log.gz 2023-11-24T09:15:31.429 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70900.log 2023-11-24T09:15:31.430 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67270.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67270.log.gz 2023-11-24T09:15:31.430 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62284.log 2023-11-24T09:15:31.431 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70900.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70900.log.gz 2023-11-24T09:15:31.431 INFO:teuthology.orchestra.run.smithi164.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39821.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61165.log 2023-11-24T09:15:31.431 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:31.432 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62284.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62284.log.gz 2023-11-24T09:15:31.432 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65662.log 2023-11-24T09:15:31.433 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43634.log 2023-11-24T09:15:31.433 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61165.log: /var/log/ceph/cluster2-client.mirror.65662.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61165.log.gz 2023-11-24T09:15:31.434 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47685.log 2023-11-24T09:15:31.435 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43634.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65662.log.gz 2023-11-24T09:15:31.439 INFO:teuthology.orchestra.run.smithi164.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43634.log.gz 2023-11-24T09:15:31.440 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52042.log 2023-11-24T09:15:31.441 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55511.log 2023-11-24T09:15:31.441 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47685.log: /var/log/ceph/cluster2-client.mirror.52042.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47685.log.gz 2023-11-24T09:15:31.441 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50980.log 2023-11-24T09:15:31.442 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52042.log.gz 2023-11-24T09:15:31.442 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55511.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55511.log.gz 2023-11-24T09:15:31.443 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39412.log 2023-11-24T09:15:31.444 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77167.log 2023-11-24T09:15:31.444 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50980.log: /var/log/ceph/cluster2-client.admin.39412.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39412.log.gz 2023-11-24T09:15:31.445 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55027.log 2023-11-24T09:15:31.445 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.77167.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.77167.log.gz 2023-11-24T09:15:31.445 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45024.log 2023-11-24T09:15:31.446 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55027.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55027.log.gz 2023-11-24T09:15:31.446 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47222.log 2023-11-24T09:15:31.447 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45024.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45024.log.gz 2023-11-24T09:15:31.448 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52400.log.gz 2023-11-24T09:15:31.448 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69754.log 2023-11-24T09:15:31.449 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.56196.log 2023-11-24T09:15:31.450 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69754.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69754.log.gz 2023-11-24T09:15:31.454 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47222.log.gz 2023-11-24T09:15:31.455 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49684.log 2023-11-24T09:15:31.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.56196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57259.log 2023-11-24T09:15:31.464 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cluster1-osd.1.log: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-11-24T09:15:31.470 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45091.log 2023-11-24T09:15:31.472 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57259.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57259.log.gz 2023-11-24T09:15:31.476 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48927.log.gz 2023-11-24T09:15:31.478 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69712.log 2023-11-24T09:15:31.479 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66726.log 2023-11-24T09:15:31.481 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69712.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69712.log.gz 2023-11-24T09:15:31.484 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45091.log.gz 2023-11-24T09:15:31.494 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48578.log 2023-11-24T09:15:31.497 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66726.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66726.log.gz 2023-11-24T09:15:31.497 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64203.log 2023-11-24T09:15:31.499 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.48578.log: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48578.log.gz 2023-11-24T09:15:31.499 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56348.log 2023-11-24T09:15:31.500 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64203.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64203.log.gz 2023-11-24T09:15:31.500 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65308.log 2023-11-24T09:15:31.502 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56348.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56348.log.gz 2023-11-24T09:15:31.502 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69588.log 2023-11-24T09:15:31.503 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65308.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65308.log.gz 2023-11-24T09:15:31.503 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68588.log 2023-11-24T09:15:31.506 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69588.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69588.log.gz 2023-11-24T09:15:31.506 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53438.log 2023-11-24T09:15:31.508 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68588.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68588.log.gz 2023-11-24T09:15:31.508 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68753.log 2023-11-24T09:15:31.509 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53438.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53438.log.gz 2023-11-24T09:15:31.509 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75812.log 2023-11-24T09:15:31.512 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68753.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68753.log.gz 2023-11-24T09:15:31.512 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64904.log 2023-11-24T09:15:31.513 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75812.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75812.log.gz 2023-11-24T09:15:31.513 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71440.log 2023-11-24T09:15:31.514 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64904.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64904.log.gz 2023-11-24T09:15:31.515 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56510.log 2023-11-24T09:15:31.515 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71440.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71440.log.gz 2023-11-24T09:15:31.516 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52358.log 2023-11-24T09:15:31.517 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56510.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56510.log.gz 2023-11-24T09:15:31.517 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51002.log 2023-11-24T09:15:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52358.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52358.log.gz 2023-11-24T09:15:31.519 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68299.log 2023-11-24T09:15:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.51002.log: 95.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51002.log.gz 2023-11-24T09:15:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46401.log 2023-11-24T09:15:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68299.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68299.log.gz 2023-11-24T09:15:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46767.log 2023-11-24T09:15:31.525 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46401.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46401.log.gz 2023-11-24T09:15:31.525 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58788.log 2023-11-24T09:15:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46767.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.56196.log.gz 2023-11-24T09:15:31.531 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46767.log.gz 2023-11-24T09:15:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50540.log 2023-11-24T09:15:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52559.log 2023-11-24T09:15:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58788.log: /var/log/ceph/cluster2-client.mirror.50540.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58788.log.gz 2023-11-24T09:15:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52321.log 2023-11-24T09:15:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52559.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52559.log.gz 2023-11-24T09:15:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50647.log 2023-11-24T09:15:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50664.log 2023-11-24T09:15:31.550 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50647.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50647.log.gz 2023-11-24T09:15:31.563 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63809.log 2023-11-24T09:15:31.564 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50664.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50664.log.gz 2023-11-24T09:15:31.578 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61596.log 2023-11-24T09:15:31.580 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63809.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63809.log.gz 2023-11-24T09:15:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45154.log 2023-11-24T09:15:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61596.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61596.log.gz 2023-11-24T09:15:31.591 INFO:teuthology.orchestra.run.smithi153.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-11-24T09:15:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74117.log 2023-11-24T09:15:31.602 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45154.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72250.log 2023-11-24T09:15:31.603 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74117.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74117.log.gz 2023-11-24T09:15:31.610 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60907.log 2023-11-24T09:15:31.611 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72250.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72250.log.gz 2023-11-24T09:15:31.614 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.45154.log.gz 2023-11-24T09:15:31.619 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61379.log 2023-11-24T09:15:31.631 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60907.log: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71418.log 2023-11-24T09:15:31.631 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61379.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53295.log.gz 2023-11-24T09:15:31.632 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61379.log.gz 2023-11-24T09:15:31.634 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60907.log.gz 2023-11-24T09:15:31.634 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73882.log 2023-11-24T09:15:31.635 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71418.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71418.log.gz 2023-11-24T09:15:31.635 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62130.log 2023-11-24T09:15:31.637 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73882.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73882.log.gz 2023-11-24T09:15:31.637 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.39840.log 2023-11-24T09:15:31.638 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62130.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62130.log.gz 2023-11-24T09:15:31.639 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75747.log 2023-11-24T09:15:31.648 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.3.39840.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45198.log.gz 2023-11-24T09:15:31.654 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.53948.log 2023-11-24T09:15:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75747.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75747.log.gz 2023-11-24T09:15:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76805.log 2023-11-24T09:15:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.53948.log: 94.1%gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-11-24T09:15:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76805.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76805.log.gz 2023-11-24T09:15:31.668 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.3.53948.log.gz 2023-11-24T09:15:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66706.log 2023-11-24T09:15:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65072.log 2023-11-24T09:15:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66706.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66706.log.gz 2023-11-24T09:15:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42423.log 2023-11-24T09:15:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65072.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65072.log.gz 2023-11-24T09:15:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70653.log 2023-11-24T09:15:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42423.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.39840.log.gz 2023-11-24T09:15:31.700 INFO:teuthology.orchestra.run.smithi164.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42423.log.gz 2023-11-24T09:15:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54562.log 2023-11-24T09:15:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70653.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70653.log.gz 2023-11-24T09:15:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54349.log 2023-11-24T09:15:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54562.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54562.log.gz 2023-11-24T09:15:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56326.log 2023-11-24T09:15:31.705 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54349.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54349.log.gz 2023-11-24T09:15:31.705 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50540.log.gz 2023-11-24T09:15:31.705 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58400.log 2023-11-24T09:15:31.705 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50763.log 2023-11-24T09:15:31.706 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58400.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58400.log.gz 2023-11-24T09:15:31.707 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56326.log.gz 2023-11-24T09:15:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49388.log 2023-11-24T09:15:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44408.log 2023-11-24T09:15:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49388.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50763.log.gz 2023-11-24T09:15:31.709 INFO:teuthology.orchestra.run.smithi164.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49388.log.gz 2023-11-24T09:15:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53900.log 2023-11-24T09:15:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73688.log 2023-11-24T09:15:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53900.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44408.log.gz 2023-11-24T09:15:31.710 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53900.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41626.log 2023-11-24T09:15:31.710 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:31.718 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56171.log 2023-11-24T09:15:31.719 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41626.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41626.log.gz 2023-11-24T09:15:31.720 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51724.log 2023-11-24T09:15:31.721 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56171.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56171.log.gz 2023-11-24T09:15:31.721 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58110.log 2023-11-24T09:15:31.722 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51724.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51724.log.gz 2023-11-24T09:15:31.722 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54625.log 2023-11-24T09:15:31.723 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58110.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58110.log.gz 2023-11-24T09:15:31.723 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44058.log 2023-11-24T09:15:31.725 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54625.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54625.log.gz 2023-11-24T09:15:31.725 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42560.log 2023-11-24T09:15:31.731 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44058.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44058.log.gz 2023-11-24T09:15:31.732 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52603.log 2023-11-24T09:15:31.733 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42560.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42560.log.gz 2023-11-24T09:15:31.733 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64464.log 2023-11-24T09:15:31.736 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52603.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49763.log.gz 2023-11-24T09:15:31.745 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49684.log.gz 2023-11-24T09:15:31.745 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60662.log 2023-11-24T09:15:31.746 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45634.log 2023-11-24T09:15:31.748 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64464.log: /var/log/ceph/cluster2-client.mirror.60662.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64464.log.gz 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60662.log.gz 2023-11-24T09:15:31.749 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:31.749 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49874.log 2023-11-24T09:15:31.749 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.74372.log 2023-11-24T09:15:31.750 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49874.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49874.log.gz 2023-11-24T09:15:31.750 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61731.log 2023-11-24T09:15:31.754 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.74372.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.74372.log.gz 2023-11-24T09:15:31.754 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67104.log 2023-11-24T09:15:31.754 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61731.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61731.log.gz 2023-11-24T09:15:31.755 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69940.log 2023-11-24T09:15:31.756 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67104.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67104.log.gz 2023-11-24T09:15:31.756 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39411.log 2023-11-24T09:15:31.757 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69940.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69940.log.gz 2023-11-24T09:15:31.758 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47465.log 2023-11-24T09:15:31.758 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39411.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39411.log.gz 2023-11-24T09:15:31.758 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79634.log 2023-11-24T09:15:31.759 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47465.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47465.log.gz 2023-11-24T09:15:31.760 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43657.log 2023-11-24T09:15:31.760 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79634.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79634.log.gz 2023-11-24T09:15:31.761 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64691.log 2023-11-24T09:15:31.769 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43657.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43657.log.gz 2023-11-24T09:15:31.769 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59146.log 2023-11-24T09:15:31.769 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64691.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64691.log.gz 2023-11-24T09:15:31.770 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44037.log 2023-11-24T09:15:31.770 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59146.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59146.log.gz 2023-11-24T09:15:31.772 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40580.log 2023-11-24T09:15:31.772 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44037.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44037.log.gz 2023-11-24T09:15:31.772 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-11-24T09:15:31.772 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40580.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40580.log.gz 2023-11-24T09:15:31.772 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46595.log 2023-11-24T09:15:31.786 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71170.log 2023-11-24T09:15:31.791 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46595.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52603.log.gz 2023-11-24T09:15:31.793 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46595.log.gz 2023-11-24T09:15:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55319.log 2023-11-24T09:15:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71170.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71170.log.gz 2023-11-24T09:15:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47485.log 2023-11-24T09:15:31.796 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55319.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55319.log.gz 2023-11-24T09:15:31.796 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68485.log 2023-11-24T09:15:31.797 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47485.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47485.log.gz 2023-11-24T09:15:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52299.log 2023-11-24T09:15:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68485.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68485.log.gz 2023-11-24T09:15:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47908.log 2023-11-24T09:15:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52299.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52299.log.gz 2023-11-24T09:15:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40521.log 2023-11-24T09:15:31.801 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47908.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47908.log.gz 2023-11-24T09:15:31.801 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62584.log 2023-11-24T09:15:31.802 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40521.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52321.log.gz 2023-11-24T09:15:31.802 INFO:teuthology.orchestra.run.smithi164.stderr: 55.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40521.log.gz 2023-11-24T09:15:31.803 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71575.log 2023-11-24T09:15:31.803 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46916.log 2023-11-24T09:15:31.804 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62584.log: /var/log/ceph/cluster1-client.mirror.71575.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71575.log.gz 2023-11-24T09:15:31.805 INFO:teuthology.orchestra.run.smithi164.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62584.log.gz 2023-11-24T09:15:31.806 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57943.log 2023-11-24T09:15:31.806 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44955.log 2023-11-24T09:15:31.806 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46916.log: /var/log/ceph/cluster1-client.mirror.57943.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57943.log.gz 2023-11-24T09:15:31.811 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46916.log.gz 2023-11-24T09:15:31.811 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69010.log 2023-11-24T09:15:31.812 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60987.log 2023-11-24T09:15:31.813 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44955.log: /var/log/ceph/cluster2-client.mirror.69010.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76678.log.gz 2023-11-24T09:15:31.813 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44955.log.gz 2023-11-24T09:15:31.813 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69010.log.gz 2023-11-24T09:15:31.814 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55419.log 2023-11-24T09:15:31.814 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47625.log 2023-11-24T09:15:31.814 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47202.log 2023-11-24T09:15:31.814 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60987.log.gz 2023-11-24T09:15:31.814 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55419.log.gz 2023-11-24T09:15:31.815 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47625.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47625.log.gz 2023-11-24T09:15:31.815 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68774.log 2023-11-24T09:15:31.816 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47202.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43547.log 2023-11-24T09:15:31.816 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45656.log 2023-11-24T09:15:31.818 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43547.log: 89.3% 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43547.log.gz 2023-11-24T09:15:31.818 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68774.log.gz 2023-11-24T09:15:31.818 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79374.log 2023-11-24T09:15:31.820 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61292.log 2023-11-24T09:15:31.820 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45656.log: /var/log/ceph/cluster2-client.admin.79374.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79374.log.gz 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45656.log.gz 2023-11-24T09:15:31.820 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:31.820 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78854.log 2023-11-24T09:15:31.820 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62215.log 2023-11-24T09:15:31.820 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61292.log: /var/log/ceph/cluster2-client.admin.78854.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78854.log.gz 2023-11-24T09:15:31.820 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61292.log.gz 2023-11-24T09:15:31.824 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59592.log 2023-11-24T09:15:31.824 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45527.log 2023-11-24T09:15:31.824 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62215.log: /var/log/ceph/cluster1-client.mirror.59592.log: 87.8% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62215.log.gz 2023-11-24T09:15:31.824 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59592.log.gz 2023-11-24T09:15:31.824 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47202.log.gz 2023-11-24T09:15:31.824 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55678.log 2023-11-24T09:15:31.824 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60215.log 2023-11-24T09:15:31.824 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45527.log: /var/log/ceph/cluster1-client.mirror.55678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62543.log 2023-11-24T09:15:31.825 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50980.log.gz 2023-11-24T09:15:31.825 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55678.log.gz 2023-11-24T09:15:31.825 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39092.log 2023-11-24T09:15:31.825 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54060.log 2023-11-24T09:15:31.825 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62543.log.gz 92.4%/var/log/ceph/cluster2-client.admin.39092.log: -- replaced with /var/log/ceph/cluster2-client.mirror.60215.log.gz 2023-11-24T09:15:31.825 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:31.825 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39092.log.gz 2023-11-24T09:15:31.825 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74650.log 2023-11-24T09:15:31.825 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56906.log 2023-11-24T09:15:31.830 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74650.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54060.log.gz 2023-11-24T09:15:31.830 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45804.log 2023-11-24T09:15:31.830 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79270.log 2023-11-24T09:15:31.830 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56906.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74650.log.gz/var/log/ceph/cluster2-client.mirror.45804.log: 2023-11-24T09:15:31.830 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56906.log.gz 2023-11-24T09:15:31.830 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58544.log 2023-11-24T09:15:31.830 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52420.log 2023-11-24T09:15:31.830 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79270.log.gz 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45804.log.gz 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58544.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45527.log.gz 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58544.log.gz 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54981.log 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53157.log 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52420.log.gz 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45134.log 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59640.log 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54981.log.gz 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53157.log.gz 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71305.log 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66058.log 2023-11-24T09:15:31.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71305.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39255.log 2023-11-24T09:15:31.832 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71305.log.gz 2023-11-24T09:15:31.832 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66058.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59640.log.gz 2023-11-24T09:15:31.832 INFO:teuthology.orchestra.run.smithi164.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66058.log.gz 2023-11-24T09:15:31.835 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51588.log 2023-11-24T09:15:31.835 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49625.log 2023-11-24T09:15:31.835 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39255.log.gz 2023-11-24T09:15:31.835 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51588.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53236.log 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63959.log 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49625.log: /var/log/ceph/cluster2-client.mirror.53236.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49625.log.gz 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53236.log.gz 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76293.log 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52220.log 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63959.log: /var/log/ceph/cluster2-client.mirror.76293.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63959.log.gz 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69237.log 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76293.log.gz 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45134.log.gz 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52220.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52220.log.gz 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67671.log 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51091.log 2023-11-24T09:15:31.836 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56464.log 2023-11-24T09:15:31.838 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69237.log.gz/var/log/ceph/cluster2-client.mirror.51091.log: 2023-11-24T09:15:31.838 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67671.log.gz 2023-11-24T09:15:31.838 INFO:teuthology.orchestra.run.smithi164.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51091.log.gz 2023-11-24T09:15:31.839 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59187.log 2023-11-24T09:15:31.839 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68340.log 2023-11-24T09:15:31.839 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59187.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56464.log.gz 2023-11-24T09:15:31.839 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70563.log 2023-11-24T09:15:31.839 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48439.log 2023-11-24T09:15:31.839 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68340.log: /var/log/ceph/cluster1-client.mirror.70563.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70563.log.gz 2023-11-24T09:15:31.841 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68340.log.gz 2023-11-24T09:15:31.841 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59187.log.gz 2023-11-24T09:15:31.841 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61864.log 2023-11-24T09:15:31.841 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54308.log 2023-11-24T09:15:31.841 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71215.log 2023-11-24T09:15:31.841 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61864.log: /var/log/ceph/cluster1-client.mirror.54308.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48439.log.gz 2023-11-24T09:15:31.841 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54308.log.gz 2023-11-24T09:15:31.841 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76170.log 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60059.log 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61864.log.gz 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71215.log.gz 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76170.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76170.log.gz 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58423.log 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45176.log 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60059.log: /var/log/ceph/cluster1-client.mirror.58423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67712.log 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45176.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58423.log.gz 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60059.log.gz 2023-11-24T09:15:31.845 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45176.log.gz 2023-11-24T09:15:31.846 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56573.log 2023-11-24T09:15:31.846 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39584.log 2023-11-24T09:15:31.846 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67712.log: /var/log/ceph/cluster2-client.mirror.56573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62451.log 2023-11-24T09:15:31.846 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.39584.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39584.log.gz 2023-11-24T09:15:31.846 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67712.log.gz 2023-11-24T09:15:31.846 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56573.log.gz 2023-11-24T09:15:31.846 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58924.log 2023-11-24T09:15:31.849 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74372.log 2023-11-24T09:15:31.849 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62451.log: /var/log/ceph/cluster2-client.mirror.58924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69671.log 2023-11-24T09:15:31.849 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74372.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62451.log.gz 2023-11-24T09:15:31.849 INFO:teuthology.orchestra.run.smithi164.stderr: 76.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74372.log.gz 2023-11-24T09:15:31.849 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39178.log 2023-11-24T09:15:31.849 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60102.log 2023-11-24T09:15:31.849 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69671.log: /var/log/ceph/cluster2-client.admin.39178.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39178.log.gz 2023-11-24T09:15:31.849 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58924.log.gz 2023-11-24T09:15:31.849 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69671.log.gz 2023-11-24T09:15:31.850 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71890.log 2023-11-24T09:15:31.850 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46466.log 2023-11-24T09:15:31.850 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60102.log: /var/log/ceph/cluster1-client.mirror.71890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50842.log 2023-11-24T09:15:31.850 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46466.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71890.log.gz 2023-11-24T09:15:31.850 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60102.log.gz 2023-11-24T09:15:31.850 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52200.log 2023-11-24T09:15:31.852 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40122.log 2023-11-24T09:15:31.852 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50842.log: /var/log/ceph/cluster2-client.mirror.52200.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50842.log.gz 2023-11-24T09:15:31.852 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79764.log 2023-11-24T09:15:31.852 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52200.log.gz 2023-11-24T09:15:31.852 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.40122.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40122.log.gz 2023-11-24T09:15:31.852 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41603.log 2023-11-24T09:15:31.852 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76255.log 2023-11-24T09:15:31.852 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79764.log: /var/log/ceph/cluster1-client.mirror.41603.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79764.log.gz 2023-11-24T09:15:31.853 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48237.log 2023-11-24T09:15:31.853 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41603.log.gz 2023-11-24T09:15:31.854 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76255.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46466.log.gz 2023-11-24T09:15:31.854 INFO:teuthology.orchestra.run.smithi164.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76255.log.gz 2023-11-24T09:15:31.855 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68505.log 2023-11-24T09:15:31.856 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54124.log 2023-11-24T09:15:31.856 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48237.log: /var/log/ceph/cluster2-client.mirror.68505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44319.log 2023-11-24T09:15:31.858 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54124.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54124.log.gz 2023-11-24T09:15:31.858 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68505.log.gz 2023-11-24T09:15:31.867 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62044.log 2023-11-24T09:15:31.867 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47665.log 2023-11-24T09:15:31.867 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62044.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62044.log.gz 2023-11-24T09:15:31.868 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43570.log 2023-11-24T09:15:31.868 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47665.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44319.log.gz 2023-11-24T09:15:31.869 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47665.log.gz 2023-11-24T09:15:31.869 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57474.log 2023-11-24T09:15:31.871 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49605.log 2023-11-24T09:15:31.871 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43570.log: /var/log/ceph/cluster1-client.mirror.57474.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43570.log.gz 2023-11-24T09:15:31.871 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57474.log.gz 2023-11-24T09:15:31.871 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62370.log 2023-11-24T09:15:31.883 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43902.log 2023-11-24T09:15:31.884 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62370.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62370.log.gz 2023-11-24T09:15:31.885 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68216.log 2023-11-24T09:15:31.886 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43902.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43902.log.gz 2023-11-24T09:15:31.886 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71508.log 2023-11-24T09:15:31.888 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68216.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68216.log.gz 2023-11-24T09:15:31.888 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.74560.log 2023-11-24T09:15:31.889 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71508.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71508.log.gz 2023-11-24T09:15:31.889 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53594.log 2023-11-24T09:15:31.897 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.74560.log: 91.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74291.log 2023-11-24T09:15:31.898 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53594.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53594.log.gz 2023-11-24T09:15:31.899 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.1.74560.log.gz 2023-11-24T09:15:31.899 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41766.log 2023-11-24T09:15:31.900 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74291.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74291.log.gz 2023-11-24T09:15:31.900 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39248.log 2023-11-24T09:15:31.901 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41766.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41766.log.gz 2023-11-24T09:15:31.901 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46701.log 2023-11-24T09:15:31.902 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39248.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39248.log.gz 2023-11-24T09:15:31.902 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75935.log 2023-11-24T09:15:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46701.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46701.log.gz 2023-11-24T09:15:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69506.log 2023-11-24T09:15:31.910 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75935.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75935.log.gz 2023-11-24T09:15:31.910 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50585.log 2023-11-24T09:15:31.911 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69506.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69506.log.gz 2023-11-24T09:15:31.911 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65288.log 2023-11-24T09:15:31.913 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50585.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50585.log.gz 2023-11-24T09:15:31.913 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.44820.log 2023-11-24T09:15:31.914 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65288.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65288.log.gz 2023-11-24T09:15:31.915 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48793.log 2023-11-24T09:15:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.44820.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45634.log.gz 2023-11-24T09:15:31.930 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49467.log 2023-11-24T09:15:31.931 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48793.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48793.log.gz 2023-11-24T09:15:31.932 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64792.log 2023-11-24T09:15:31.932 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49467.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49467.log.gz 2023-11-24T09:15:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50864.log 2023-11-24T09:15:31.934 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64792.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64792.log.gz 2023-11-24T09:15:31.935 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66856.log 2023-11-24T09:15:31.946 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50864.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.44820.log.gz 2023-11-24T09:15:31.946 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63346.log 2023-11-24T09:15:31.949 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66856.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66856.log.gz 2023-11-24T09:15:31.949 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53459.log 2023-11-24T09:15:31.950 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63346.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63346.log.gz 2023-11-24T09:15:31.950 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64134.log 2023-11-24T09:15:31.951 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53459.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53459.log.gz 2023-11-24T09:15:31.951 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60929.log 2023-11-24T09:15:31.952 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64134.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64134.log.gz 2023-11-24T09:15:31.953 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58377.log 2023-11-24T09:15:31.954 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60929.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60929.log.gz 2023-11-24T09:15:31.954 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73968.log 2023-11-24T09:15:31.955 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58377.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58377.log.gz 2023-11-24T09:15:31.957 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45590.log 2023-11-24T09:15:31.957 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73968.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73968.log.gz 2023-11-24T09:15:31.957 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67022.log 2023-11-24T09:15:31.963 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45590.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45590.log.gz 2023-11-24T09:15:31.963 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45001.log 2023-11-24T09:15:31.964 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67022.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67022.log.gz 2023-11-24T09:15:31.964 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.67312.log 2023-11-24T09:15:31.965 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45001.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45001.log.gz 2023-11-24T09:15:31.966 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79686.log 2023-11-24T09:15:31.978 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.67312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77554.log 2023-11-24T09:15:31.979 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79686.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79686.log.gz 2023-11-24T09:15:31.994 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70180.log 2023-11-24T09:15:31.995 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77554.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77554.log.gz 2023-11-24T09:15:31.995 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42766.log 2023-11-24T09:15:31.996 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70180.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70180.log.gz 2023-11-24T09:15:32.017 INFO:teuthology.orchestra.run.smithi164.stderr: 92.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78464.log 2023-11-24T09:15:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.42766.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.42766.log.gz 2023-11-24T09:15:32.017 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.67312.log.gz 2023-11-24T09:15:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43004.log 2023-11-24T09:15:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78464.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78464.log.gz 2023-11-24T09:15:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66752.log 2023-11-24T09:15:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43004.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43004.log.gz 2023-11-24T09:15:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67836.log 2023-11-24T09:15:32.021 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66752.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66752.log.gz 2023-11-24T09:15:32.021 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65797.log 2023-11-24T09:15:32.022 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67836.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67836.log.gz 2023-11-24T09:15:32.022 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74765.log 2023-11-24T09:15:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65797.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65797.log.gz 2023-11-24T09:15:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77125.log 2023-11-24T09:15:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74765.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74765.log.gz 2023-11-24T09:15:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75153.log 2023-11-24T09:15:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.77125.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77125.log.gz 2023-11-24T09:15:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63520.log 2023-11-24T09:15:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75153.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75153.log.gz 2023-11-24T09:15:32.028 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71598.log 2023-11-24T09:15:32.029 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63520.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63520.log.gz 2023-11-24T09:15:32.029 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59503.log 2023-11-24T09:15:32.030 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71598.log: 84.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71598.log.gz 2023-11-24T09:15:32.031 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73688.log.gz 2023-11-24T09:15:32.031 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79868.log 2023-11-24T09:15:32.031 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.45656.log 2023-11-24T09:15:32.032 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59503.log: /var/log/ceph/cluster2-client.admin.79868.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79868.log.gz 2023-11-24T09:15:32.032 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59503.log.gz 2023-11-24T09:15:32.032 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72093.log 2023-11-24T09:15:32.032 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59729.log 2023-11-24T09:15:32.033 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.45656.log: /var/log/ceph/cluster1-client.mirror.72093.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72093.log.gz 2023-11-24T09:15:32.038 INFO:teuthology.orchestra.run.smithi164.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.45656.log.gz 2023-11-24T09:15:32.039 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56487.log 2023-11-24T09:15:32.039 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77528.log 2023-11-24T09:15:32.040 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59729.log: /var/log/ceph/cluster1-client.mirror.56487.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59729.log.gz 2023-11-24T09:15:32.040 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56487.log.gz 2023-11-24T09:15:32.040 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69216.log 2023-11-24T09:15:32.041 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45460.log 2023-11-24T09:15:32.041 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77528.log: /var/log/ceph/cluster2-client.mirror.69216.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77528.log.gz 2023-11-24T09:15:32.041 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61800.log 2023-11-24T09:15:32.041 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45460.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45460.log.gz 2023-11-24T09:15:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39040.log 2023-11-24T09:15:32.043 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61800.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69216.log.gz 2023-11-24T09:15:32.043 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61800.log.gz 2023-11-24T09:15:32.044 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78074.log 2023-11-24T09:15:32.044 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55724.log 2023-11-24T09:15:32.044 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39040.log: /var/log/ceph/cluster2-client.admin.78074.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39040.log.gz 2023-11-24T09:15:32.044 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78074.log.gz 2023-11-24T09:15:32.044 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58567.log 2023-11-24T09:15:32.045 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49047.log 2023-11-24T09:15:32.045 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55724.log: /var/log/ceph/cluster1-client.mirror.58567.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55724.log.gz 2023-11-24T09:15:32.045 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58567.log.gz 2023-11-24T09:15:32.046 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57301.log 2023-11-24T09:15:32.047 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53948.log 2023-11-24T09:15:32.047 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49047.log: /var/log/ceph/cluster2-client.mirror.57301.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49047.log.gz 2023-11-24T09:15:32.047 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71710.log 2023-11-24T09:15:32.047 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53948.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57301.log.gz 2023-11-24T09:15:32.048 INFO:teuthology.orchestra.run.smithi164.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53948.log.gz 2023-11-24T09:15:32.048 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63478.log 2023-11-24T09:15:32.049 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43068.log 2023-11-24T09:15:32.049 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71710.log: /var/log/ceph/cluster2-client.mirror.63478.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71710.log.gz 2023-11-24T09:15:32.050 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64645.log 2023-11-24T09:15:32.050 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7%/var/log/ceph/cluster2-client.mirror.43068.log: -- replaced with /var/log/ceph/cluster2-client.mirror.63478.log.gz 2023-11-24T09:15:32.055 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43068.log.gz 2023-11-24T09:15:32.055 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55050.log 2023-11-24T09:15:32.055 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58636.log 2023-11-24T09:15:32.056 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64645.log: /var/log/ceph/cluster1-client.mirror.55050.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64645.log.gz 2023-11-24T09:15:32.056 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55050.log.gz 2023-11-24T09:15:32.056 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77340.log 2023-11-24T09:15:32.057 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58636.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58636.log.gz 2023-11-24T09:15:32.057 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47565.log 2023-11-24T09:15:32.058 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59367.log 2023-11-24T09:15:32.058 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.77340.log: /var/log/ceph/cluster2-client.mirror.47565.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77340.log.gz 2023-11-24T09:15:32.059 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49027.log 2023-11-24T09:15:32.059 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7%/var/log/ceph/cluster1-client.mirror.59367.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47565.log.gz 2023-11-24T09:15:32.059 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59367.log.gz 2023-11-24T09:15:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47179.log 2023-11-24T09:15:32.060 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54675.log 2023-11-24T09:15:32.066 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49027.log: /var/log/ceph/cluster1-client.mirror.47179.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47179.log.gz 2023-11-24T09:15:32.070 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77240.log 2023-11-24T09:15:32.072 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54675.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54675.log.gz 2023-11-24T09:15:32.072 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58966.log 2023-11-24T09:15:32.073 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.77240.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77240.log.gz 2023-11-24T09:15:32.073 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51128.log 2023-11-24T09:15:32.074 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58966.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58966.log.gz 2023-11-24T09:15:32.075 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60235.log 2023-11-24T09:15:32.075 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51128.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51128.log.gz 2023-11-24T09:15:32.076 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49891.log 2023-11-24T09:15:32.078 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60235.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60235.log.gz 2023-11-24T09:15:32.078 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59275.log 2023-11-24T09:15:32.080 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49891.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49891.log.gz 2023-11-24T09:15:32.080 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44477.log 2023-11-24T09:15:32.081 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59275.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59275.log.gz 2023-11-24T09:15:32.081 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64023.log 2023-11-24T09:15:32.082 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44477.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44477.log.gz 2023-11-24T09:15:32.083 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78698.log 2023-11-24T09:15:32.085 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64023.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64023.log.gz 2023-11-24T09:15:32.085 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74850.log 2023-11-24T09:15:32.086 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78698.log: 28.2% -- replaced with /var/log/ceph/cluster2-client.admin.78698.log.gz 2023-11-24T09:15:32.086 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71688.log 2023-11-24T09:15:32.089 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74850.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74850.log.gz 2023-11-24T09:15:32.089 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40560.log 2023-11-24T09:15:32.090 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71688.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71688.log.gz 2023-11-24T09:15:32.090 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77265.log 2023-11-24T09:15:32.092 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40560.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40560.log.gz 2023-11-24T09:15:32.093 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70878.log 2023-11-24T09:15:32.093 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.77265.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77265.log.gz 2023-11-24T09:15:32.093 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75217.log 2023-11-24T09:15:32.094 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70878.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70878.log.gz 2023-11-24T09:15:32.094 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72160.log 2023-11-24T09:15:32.095 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75217.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75217.log.gz 2023-11-24T09:15:32.095 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70428.log 2023-11-24T09:15:32.096 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72160.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72160.log.gz 2023-11-24T09:15:32.097 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49106.log 2023-11-24T09:15:32.097 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70428.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70428.log.gz 2023-11-24T09:15:32.098 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69774.log 2023-11-24T09:15:32.110 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61033.log 2023-11-24T09:15:32.113 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69774.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69774.log.gz 2023-11-24T09:15:32.126 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79400.log 2023-11-24T09:15:32.127 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61033.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61033.log.gz 2023-11-24T09:15:32.127 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42981.log 2023-11-24T09:15:32.128 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79400.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79400.log.gz 2023-11-24T09:15:32.137 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50864.log.gz 2023-11-24T09:15:32.146 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47093.log 2023-11-24T09:15:32.147 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70450.log 2023-11-24T09:15:32.147 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42981.log: /var/log/ceph/cluster1-client.mirror.47093.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42981.log.gz 2023-11-24T09:15:32.152 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47093.log.gz 2023-11-24T09:15:32.153 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71778.log 2023-11-24T09:15:32.153 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70450.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70450.log.gz 2023-11-24T09:15:32.154 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57987.log 2023-11-24T09:15:32.155 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71778.log: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71778.log.gz 2023-11-24T09:15:32.155 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67588.log 2023-11-24T09:15:32.157 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57987.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57987.log.gz 2023-11-24T09:15:32.157 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47605.log 2023-11-24T09:15:32.159 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67588.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67588.log.gz 2023-11-24T09:15:32.159 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69030.log 2023-11-24T09:15:32.162 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47605.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47605.log.gz 2023-11-24T09:15:32.162 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65595.log 2023-11-24T09:15:32.164 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69030.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69030.log.gz 2023-11-24T09:15:32.164 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66662.log 2023-11-24T09:15:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65595.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65595.log.gz 2023-11-24T09:15:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78256.log 2023-11-24T09:15:32.166 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66662.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66662.log.gz 2023-11-24T09:15:32.166 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49642.log 2023-11-24T09:15:32.167 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78256.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78256.log.gz 2023-11-24T09:15:32.167 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.51002.log 2023-11-24T09:15:32.168 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49642.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49642.log.gz 2023-11-24T09:15:32.169 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70270.log 2023-11-24T09:15:32.182 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.2.51002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68815.log 2023-11-24T09:15:32.183 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70270.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70270.log.gz 2023-11-24T09:15:32.190 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49605.log.gz 2023-11-24T09:15:32.198 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47525.log 2023-11-24T09:15:32.199 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57061.log 2023-11-24T09:15:32.200 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68815.log: /var/log/ceph/cluster2-client.mirror.47525.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47525.log.gz 2023-11-24T09:15:32.200 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51935.log 2023-11-24T09:15:32.201 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57061.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68815.log.gz 2023-11-24T09:15:32.201 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57061.log.gz 2023-11-24T09:15:32.202 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77944.log 2023-11-24T09:15:32.204 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51935.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51935.log.gz 2023-11-24T09:15:32.204 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39842.log 2023-11-24T09:15:32.204 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77944.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77944.log.gz 2023-11-24T09:15:32.204 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76276.log 2023-11-24T09:15:32.205 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.39842.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39842.log.gz 2023-11-24T09:15:32.205 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56708.log 2023-11-24T09:15:32.206 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76276.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76276.log.gz 2023-11-24T09:15:32.206 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75955.log 2023-11-24T09:15:32.207 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56708.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56708.log.gz 2023-11-24T09:15:32.208 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62067.log 2023-11-24T09:15:32.208 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75955.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75955.log.gz 2023-11-24T09:15:32.209 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72273.log 2023-11-24T09:15:32.210 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62067.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62067.log.gz 2023-11-24T09:15:32.210 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65250.log 2023-11-24T09:15:32.211 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72273.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72273.log.gz 2023-11-24T09:15:32.211 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47785.log 2023-11-24T09:15:32.213 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65250.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65250.log.gz 2023-11-24T09:15:32.213 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57146.log 2023-11-24T09:15:32.214 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47785.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47785.log.gz 2023-11-24T09:15:32.214 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42300.log 2023-11-24T09:15:32.215 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57146.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57146.log.gz 2023-11-24T09:15:32.216 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70002.log 2023-11-24T09:15:32.221 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42300.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42300.log.gz 2023-11-24T09:15:32.222 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59051.log 2023-11-24T09:15:32.223 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.70002.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70002.log.gz 2023-11-24T09:15:32.223 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71845.log 2023-11-24T09:15:32.224 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59051.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59051.log.gz 2023-11-24T09:15:32.224 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43478.log 2023-11-24T09:15:32.225 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71845.log: 84.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71845.log.gz 2023-11-24T09:15:32.225 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.76989.log 2023-11-24T09:15:32.226 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43478.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43478.log.gz 2023-11-24T09:15:32.227 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60820.log 2023-11-24T09:15:32.227 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.2.76989.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49027.log.gz 2023-11-24T09:15:32.228 INFO:teuthology.orchestra.run.smithi164.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.76989.log.gz 2023-11-24T09:15:32.229 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72436.log 2023-11-24T09:15:32.229 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70383.log 2023-11-24T09:15:32.230 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60820.log: /var/log/ceph/cluster2-client.mirror.72436.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60820.log.gz 2023-11-24T09:15:32.230 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60887.log 2023-11-24T09:15:32.231 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70383.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70383.log.gz 2023-11-24T09:15:32.231 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79842.log 2023-11-24T09:15:32.233 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60887.log: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.51002.log.gz 2023-11-24T09:15:32.233 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60887.log.gz 2023-11-24T09:15:32.234 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48215.log 2023-11-24T09:15:32.234 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51298.log 2023-11-24T09:15:32.235 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79842.log: /var/log/ceph/cluster2-client.mirror.48215.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79842.log.gz 2023-11-24T09:15:32.235 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34754.log 2023-11-24T09:15:32.235 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48215.log.gz 2023-11-24T09:15:32.235 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51298.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51298.log.gz 2023-11-24T09:15:32.236 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61123.log 2023-11-24T09:15:32.236 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67228.log 2023-11-24T09:15:32.236 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.34754.log: /var/log/ceph/cluster2-client.mirror.61123.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34754.log.gz 2023-11-24T09:15:32.237 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76492.log 2023-11-24T09:15:32.238 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67228.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61123.log.gz 2023-11-24T09:15:32.239 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67228.log.gz 2023-11-24T09:15:32.240 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73903.log 2023-11-24T09:15:32.240 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57215.log 2023-11-24T09:15:32.241 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76492.log: /var/log/ceph/cluster1-client.mirror.73903.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.73903.log.gz 2023-11-24T09:15:32.241 INFO:teuthology.orchestra.run.smithi164.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76492.log.gz 2023-11-24T09:15:32.242 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48736.log 2023-11-24T09:15:32.242 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62192.log 2023-11-24T09:15:32.243 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57215.log: /var/log/ceph/cluster2-client.mirror.48736.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57215.log.gz 2023-11-24T09:15:32.243 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69072.log 2023-11-24T09:15:32.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62192.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62192.log.gz 2023-11-24T09:15:32.244 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67359.log 2023-11-24T09:15:32.245 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69072.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69072.log.gz 2023-11-24T09:15:32.245 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.75425.log 2023-11-24T09:15:32.246 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67359.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67359.log.gz 2023-11-24T09:15:32.247 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55810.log 2023-11-24T09:15:32.250 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.3.75425.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48237.log.gz 2023-11-24T09:15:32.254 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75399.log 2023-11-24T09:15:32.256 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55810.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55810.log.gz 2023-11-24T09:15:32.256 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50016.log 2023-11-24T09:15:32.258 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75399.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75399.log.gz 2023-11-24T09:15:32.258 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70518.log 2023-11-24T09:15:32.262 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50016.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51588.log.gz 2023-11-24T09:15:32.267 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-11-24T09:15:32.267 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70518.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70518.log.gz 2023-11-24T09:15:32.268 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48307.log 2023-11-24T09:15:32.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2.audit.log: 92.6% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-11-24T09:15:32.272 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53194.log 2023-11-24T09:15:32.273 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48307.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48307.log.gz 2023-11-24T09:15:32.273 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74046.log 2023-11-24T09:15:32.274 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53194.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53194.log.gz 2023-11-24T09:15:32.275 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43593.log 2023-11-24T09:15:32.276 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74046.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74046.log.gz 2023-11-24T09:15:32.276 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.41904.log 2023-11-24T09:15:32.277 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43593.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43593.log.gz 2023-11-24T09:15:32.277 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68733.log 2023-11-24T09:15:32.283 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.41904.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.1.41904.log.gz 2023-11-24T09:15:32.283 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40098.log 2023-11-24T09:15:32.284 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68733.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68733.log.gz 2023-11-24T09:15:32.284 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66458.log 2023-11-24T09:15:32.285 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.40098.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40098.log.gz 2023-11-24T09:15:32.285 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72414.log 2023-11-24T09:15:32.287 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66458.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66458.log.gz 2023-11-24T09:15:32.288 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68030.log 2023-11-24T09:15:32.289 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.72414.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72414.log.gz 2023-11-24T09:15:32.289 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77215.log 2023-11-24T09:15:32.290 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68030.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68030.log.gz 2023-11-24T09:15:32.291 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46896.log 2023-11-24T09:15:32.291 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.77215.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77215.log.gz 2023-11-24T09:15:32.291 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49218.log 2023-11-24T09:15:32.297 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46896.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46896.log.gz 2023-11-24T09:15:32.298 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62408.log 2023-11-24T09:15:32.298 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49218.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49218.log.gz 2023-11-24T09:15:32.299 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56639.log 2023-11-24T09:15:32.300 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62408.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62408.log.gz 2023-11-24T09:15:32.301 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76852.log 2023-11-24T09:15:32.301 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56639.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56639.log.gz 2023-11-24T09:15:32.302 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45506.log 2023-11-24T09:15:32.302 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76852.log: 83.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76852.log.gz 2023-11-24T09:15:32.303 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56102.log 2023-11-24T09:15:32.304 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45506.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45506.log.gz 2023-11-24T09:15:32.304 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66939.log 2023-11-24T09:15:32.305 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56102.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56102.log.gz 2023-11-24T09:15:32.305 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46156.log 2023-11-24T09:15:32.307 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66939.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66939.log.gz 2023-11-24T09:15:32.307 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40436.log 2023-11-24T09:15:32.308 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46156.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46156.log.gz 2023-11-24T09:15:32.308 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-11-24T09:15:32.309 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40436.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40436.log.gz 2023-11-24T09:15:32.309 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64881.log 2023-11-24T09:15:32.315 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2.log: 87.9% -- replaced with /var/log/ceph/cluster2.log.gz 2023-11-24T09:15:32.315 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52262.log 2023-11-24T09:15:32.316 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64881.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64881.log.gz 2023-11-24T09:15:32.317 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71980.log 2023-11-24T09:15:32.317 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52262.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52262.log.gz 2023-11-24T09:15:32.318 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52341.log 2023-11-24T09:15:32.318 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71980.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71980.log.gz 2023-11-24T09:15:32.319 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79166.log 2023-11-24T09:15:32.319 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52341.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52341.log.gz 2023-11-24T09:15:32.320 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70608.log 2023-11-24T09:15:32.320 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79166.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79166.log.gz 2023-11-24T09:15:32.321 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57103.log 2023-11-24T09:15:32.321 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70608.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70608.log.gz 2023-11-24T09:15:32.322 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60148.log 2023-11-24T09:15:32.323 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57103.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57103.log.gz 2023-11-24T09:15:32.324 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62664.log 2023-11-24T09:15:32.324 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60148.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60148.log.gz 2023-11-24T09:15:32.325 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75680.log 2023-11-24T09:15:32.326 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62664.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62664.log.gz 2023-11-24T09:15:32.326 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62733.log 2023-11-24T09:15:32.347 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61975.log 2023-11-24T09:15:32.348 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62733.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62733.log.gz 2023-11-24T09:15:32.348 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57123.log 2023-11-24T09:15:32.349 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61975.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61975.log.gz 2023-11-24T09:15:32.349 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69981.log 2023-11-24T09:15:32.350 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57123.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57123.log.gz 2023-11-24T09:15:32.363 INFO:teuthology.orchestra.run.smithi164.stderr: 95.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.75425.log.gz 2023-11-24T09:15:32.366 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67187.log 2023-11-24T09:15:32.368 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69981.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69981.log.gz 2023-11-24T09:15:32.368 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51397.log 2023-11-24T09:15:32.369 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67187.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67187.log.gz 2023-11-24T09:15:32.370 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47765.log 2023-11-24T09:15:32.370 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51397.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51397.log.gz 2023-11-24T09:15:32.371 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75378.log 2023-11-24T09:15:32.372 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47765.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47765.log.gz 2023-11-24T09:15:32.372 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68361.log 2023-11-24T09:15:32.373 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75378.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75378.log.gz 2023-11-24T09:15:32.374 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51150.log 2023-11-24T09:15:32.374 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68361.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68361.log.gz 2023-11-24T09:15:32.375 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70945.log 2023-11-24T09:15:32.390 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40180.log 2023-11-24T09:15:32.391 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70945.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70945.log.gz 2023-11-24T09:15:32.406 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65820.log 2023-11-24T09:15:32.407 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.40180.log: 67.2% -- replaced with /var/log/ceph/cluster1-client.admin.40180.log.gz 2023-11-24T09:15:32.416 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50016.log.gz 2023-11-24T09:15:32.416 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78490.log 2023-11-24T09:15:32.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65820.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65820.log.gz 2023-11-24T09:15:32.418 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56125.log 2023-11-24T09:15:32.419 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78490.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78490.log.gz 2023-11-24T09:15:32.419 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55488.log 2023-11-24T09:15:32.419 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9%/var/log/ceph/cluster1-client.mirror.56125.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49106.log.gz 2023-11-24T09:15:32.420 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56125.log.gz 2023-11-24T09:15:32.425 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78958.log 2023-11-24T09:15:32.425 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74250.log 2023-11-24T09:15:32.425 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55488.log: /var/log/ceph/cluster2-client.admin.78958.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78958.log.gz 2023-11-24T09:15:32.425 INFO:teuthology.orchestra.run.smithi164.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55488.log.gz 2023-11-24T09:15:32.426 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59927.log 2023-11-24T09:15:32.426 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69444.log 2023-11-24T09:15:32.426 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74250.log: /var/log/ceph/cluster1-client.mirror.59927.log: 87.7% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74250.log.gz 2023-11-24T09:15:32.426 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59927.log.gz 2023-11-24T09:15:32.426 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41430.log 2023-11-24T09:15:32.427 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51509.log 2023-11-24T09:15:32.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69444.log: /var/log/ceph/cluster2-client.mirror.41430.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69444.log.gz 2023-11-24T09:15:32.427 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58944.log 2023-11-24T09:15:32.427 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41430.log.gz/var/log/ceph/cluster2-client.mirror.51509.log: 2023-11-24T09:15:32.438 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58067.log 2023-11-24T09:15:32.441 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58944.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58944.log.gz 2023-11-24T09:15:32.441 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39482.log 2023-11-24T09:15:32.443 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58067.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58067.log.gz 2023-11-24T09:15:32.443 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71958.log 2023-11-24T09:15:32.444 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39482.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39482.log.gz 2023-11-24T09:15:32.444 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70203.log 2023-11-24T09:15:32.445 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71958.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71958.log.gz 2023-11-24T09:15:32.445 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62710.log 2023-11-24T09:15:32.446 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70203.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70203.log.gz 2023-11-24T09:15:32.446 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74161.log 2023-11-24T09:15:32.447 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62710.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62710.log.gz 2023-11-24T09:15:32.447 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39118.log 2023-11-24T09:15:32.449 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74161.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74161.log.gz 2023-11-24T09:15:32.449 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73646.log 2023-11-24T09:15:32.450 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39118.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39118.log.gz 2023-11-24T09:15:32.450 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59615.log 2023-11-24T09:15:32.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73646.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73646.log.gz 2023-11-24T09:15:32.452 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60526.log 2023-11-24T09:15:32.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59615.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59615.log.gz 2023-11-24T09:15:32.453 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68877.log 2023-11-24T09:15:32.454 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60526.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60526.log.gz 2023-11-24T09:15:32.455 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52870.log 2023-11-24T09:15:32.457 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68877.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68877.log.gz 2023-11-24T09:15:32.458 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46939.log 2023-11-24T09:15:32.458 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72436.log.gz 2023-11-24T09:15:32.459 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52870.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52870.log.gz 2023-11-24T09:15:32.459 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50249.log 2023-11-24T09:15:32.459 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62474.log 2023-11-24T09:15:32.460 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46939.log: /var/log/ceph/cluster2-client.mirror.50249.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50249.log.gz 2023-11-24T09:15:32.464 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46939.log.gz 2023-11-24T09:15:32.465 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51261.log 2023-11-24T09:15:32.466 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75425.log 2023-11-24T09:15:32.466 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62474.log: /var/log/ceph/cluster2-client.mirror.51261.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51261.log.gz 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62474.log.gz 2023-11-24T09:15:32.466 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:32.467 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59881.log 2023-11-24T09:15:32.468 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61359.log 2023-11-24T09:15:32.468 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75425.log: 81.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75425.log.gz 2023-11-24T09:15:32.468 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74809.log 2023-11-24T09:15:32.468 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59881.log.gz 2023-11-24T09:15:32.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61359.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61359.log.gz 2023-11-24T09:15:32.470 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52141.log 2023-11-24T09:15:32.470 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78178.log 2023-11-24T09:15:32.471 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74809.log: /var/log/ceph/cluster2-client.mirror.52141.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74809.log.gz 2023-11-24T09:15:32.471 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52141.log.gz 2023-11-24T09:15:32.471 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58486.log 2023-11-24T09:15:32.472 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58613.log 2023-11-24T09:15:32.472 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78178.log: /var/log/ceph/cluster2-client.mirror.58486.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78178.log.gz 2023-11-24T09:15:32.473 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77814.log 2023-11-24T09:15:32.473 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58613.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58486.log.gz 2023-11-24T09:15:32.473 INFO:teuthology.orchestra.run.smithi164.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58613.log.gz 2023-11-24T09:15:32.474 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73367.log 2023-11-24T09:15:32.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39684.log 2023-11-24T09:15:32.475 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77814.log: /var/log/ceph/cluster1-client.mirror.73367.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77814.log.gz 2023-11-24T09:15:32.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79114.log 2023-11-24T09:15:32.475 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73367.log.gz 2023-11-24T09:15:32.475 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39684.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39684.log.gz 2023-11-24T09:15:32.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55229.log 2023-11-24T09:15:32.476 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57495.log 2023-11-24T09:15:32.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79114.log: /var/log/ceph/cluster1-client.mirror.55229.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79114.log.gz 2023-11-24T09:15:32.477 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77918.log 2023-11-24T09:15:32.477 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55229.log.gz 2023-11-24T09:15:32.477 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57495.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57495.log.gz 2023-11-24T09:15:32.478 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76636.log 2023-11-24T09:15:32.479 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56418.log 2023-11-24T09:15:32.479 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77918.log: /var/log/ceph/cluster2-client.mirror.76636.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77918.log.gz 2023-11-24T09:15:32.479 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40786.log 2023-11-24T09:15:32.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56418.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76636.log.gz 2023-11-24T09:15:32.480 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56418.log.gz 2023-11-24T09:15:32.480 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63586.log 2023-11-24T09:15:32.480 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61056.log 2023-11-24T09:15:32.481 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40786.log: /var/log/ceph/cluster1-client.mirror.63586.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63586.log.gz 2023-11-24T09:15:32.494 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79322.log 2023-11-24T09:15:32.495 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61056.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61056.log.gz 2023-11-24T09:15:32.496 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77102.log 2023-11-24T09:15:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79322.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79322.log.gz 2023-11-24T09:15:32.497 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40146.log 2023-11-24T09:15:32.498 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.77102.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.77102.log.gz 2023-11-24T09:15:32.498 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42690.log 2023-11-24T09:15:32.499 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.40146.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.40146.log.gz 2023-11-24T09:15:32.499 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40163.log 2023-11-24T09:15:32.505 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42690.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42690.log.gz 2023-11-24T09:15:32.505 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39734.log 2023-11-24T09:15:32.506 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.40163.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.40163.log.gz 2023-11-24T09:15:32.507 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43524.log 2023-11-24T09:15:32.507 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39734.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39734.log.gz 2023-11-24T09:15:32.508 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50706.log 2023-11-24T09:15:32.508 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43524.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43524.log.gz 2023-11-24T09:15:32.509 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65095.log 2023-11-24T09:15:32.518 INFO:teuthology.orchestra.run.smithi153.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-11-24T09:15:32.522 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50706.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75680.log.gz 2023-11-24T09:15:32.522 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50498.log 2023-11-24T09:15:32.523 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68195.log 2023-11-24T09:15:32.523 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65095.log: /var/log/ceph/cluster2-client.mirror.50498.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65095.log.gz 2023-11-24T09:15:32.524 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50498.log.gz 2023-11-24T09:15:32.525 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69857.log 2023-11-24T09:15:32.527 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68195.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68195.log.gz 2023-11-24T09:15:32.528 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43267.log 2023-11-24T09:15:32.529 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69857.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69857.log.gz 2023-11-24T09:15:32.529 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58223.log 2023-11-24T09:15:32.530 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.43267.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43267.log.gz 2023-11-24T09:15:32.530 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57966.log 2023-11-24T09:15:32.533 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58223.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58223.log.gz 2023-11-24T09:15:32.533 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44454.log 2023-11-24T09:15:32.534 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57966.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57966.log.gz 2023-11-24T09:15:32.534 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69547.log 2023-11-24T09:15:32.535 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44454.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44454.log.gz 2023-11-24T09:15:32.535 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55534.log 2023-11-24T09:15:32.537 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69547.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69547.log.gz 2023-11-24T09:15:32.537 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54958.log 2023-11-24T09:15:32.538 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55534.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55534.log.gz 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51150.log.gz 2023-11-24T09:15:32.538 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:32.538 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56929.log 2023-11-24T09:15:32.539 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39144.log 2023-11-24T09:15:32.540 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54958.log: /var/log/ceph/cluster1-client.mirror.56929.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54958.log.gz 2023-11-24T09:15:32.540 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56929.log.gz 2023-11-24T09:15:32.540 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73629.log 2023-11-24T09:15:32.541 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39784.log 2023-11-24T09:15:32.541 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39144.log: /var/log/ceph/cluster2-client.mirror.73629.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39144.log.gz 2023-11-24T09:15:32.541 INFO:teuthology.orchestra.run.smithi164.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73629.log.gz 2023-11-24T09:15:32.541 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60013.log 2023-11-24T09:15:32.542 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65572.log 2023-11-24T09:15:32.542 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39784.log: /var/log/ceph/cluster2-client.mirror.60013.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39784.log.gz 2023-11-24T09:15:32.542 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57649.log 2023-11-24T09:15:32.542 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60013.log.gz 2023-11-24T09:15:32.543 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65572.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65572.log.gz 2023-11-24T09:15:32.543 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58659.log 2023-11-24T09:15:32.543 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66616.log 2023-11-24T09:15:32.544 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57649.log: /var/log/ceph/cluster1-client.mirror.58659.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57649.log.gz 2023-11-24T09:15:32.544 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58659.log.gz 2023-11-24T09:15:32.544 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60843.log 2023-11-24T09:15:32.545 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66616.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66616.log.gz 2023-11-24T09:15:32.545 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75515.log 2023-11-24T09:15:32.546 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61575.log 2023-11-24T09:15:32.546 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60843.log: /var/log/ceph/cluster1-client.mirror.75515.log: 86.9% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60843.log.gz 2023-11-24T09:15:32.546 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.75515.log.gz 2023-11-24T09:15:32.546 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42820.log 2023-11-24T09:15:32.547 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51746.log 2023-11-24T09:15:32.547 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61575.log: /var/log/ceph/cluster2-client.mirror.42820.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61575.log.gz 2023-11-24T09:15:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41406.log 2023-11-24T09:15:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51746.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42820.log.gz 2023-11-24T09:15:32.552 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48736.log.gz 2023-11-24T09:15:32.562 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41324.log 2023-11-24T09:15:32.563 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56731.log 2023-11-24T09:15:32.563 INFO:teuthology.orchestra.run.smithi164.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41406.log.gz/var/log/ceph/cluster1-client.admin.41324.log: 2023-11-24T09:15:32.564 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70720.log 2023-11-24T09:15:32.564 INFO:teuthology.orchestra.run.smithi164.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.41324.log.gz 2023-11-24T09:15:32.565 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56731.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56731.log.gz 2023-11-24T09:15:32.565 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53216.log 2023-11-24T09:15:32.565 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52700.log 2023-11-24T09:15:32.566 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70720.log.gz 2023-11-24T09:15:32.567 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62931.log 2023-11-24T09:15:32.567 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52700.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52700.log.gz 2023-11-24T09:15:32.567 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40248.log 2023-11-24T09:15:32.570 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62931.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62931.log.gz 2023-11-24T09:15:32.570 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44365.log 2023-11-24T09:15:32.570 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.40248.log: 57.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40248.log.gz 2023-11-24T09:15:32.570 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78282.log 2023-11-24T09:15:32.571 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44365.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44365.log.gz 2023-11-24T09:15:32.572 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.75217.log 2023-11-24T09:15:32.572 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78282.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78282.log.gz 2023-11-24T09:15:32.572 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69299.log 2023-11-24T09:15:32.579 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.75217.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77840.log 2023-11-24T09:15:32.579 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69299.log: -- replaced with /var/log/ceph/cluster2-client.mirror.0.75217.log.gz 2023-11-24T09:15:32.580 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69299.log.gz 2023-11-24T09:15:32.580 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77684.log 2023-11-24T09:15:32.581 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77840.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77840.log.gz 2023-11-24T09:15:32.581 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67567.log 2023-11-24T09:15:32.582 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77684.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77684.log.gz 2023-11-24T09:15:32.582 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51360.log 2023-11-24T09:15:32.585 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67567.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67567.log.gz 2023-11-24T09:15:32.585 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57831.log 2023-11-24T09:15:32.586 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51360.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51360.log.gz 2023-11-24T09:15:32.586 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76448.log 2023-11-24T09:15:32.588 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57831.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57831.log.gz 2023-11-24T09:15:32.588 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70698.log 2023-11-24T09:15:32.589 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76448.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.76448.log.gz 2023-11-24T09:15:32.589 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31010.log 2023-11-24T09:15:32.590 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70698.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70698.log.gz 2023-11-24T09:15:32.590 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46529.log 2023-11-24T09:15:32.591 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.tmp-client.admin.31010.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31010.log.gz 2023-11-24T09:15:32.591 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75897.log 2023-11-24T09:15:32.597 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46529.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46529.log.gz 2023-11-24T09:15:32.598 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66437.log 2023-11-24T09:15:32.600 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75897.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75897.log.gz 2023-11-24T09:15:32.601 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42958.log 2023-11-24T09:15:32.602 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66437.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66437.log.gz 2023-11-24T09:15:32.602 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74270.log 2023-11-24T09:15:32.603 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42958.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42958.log.gz 2023-11-24T09:15:32.603 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77788.log 2023-11-24T09:15:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74270.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74270.log.gz 2023-11-24T09:15:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56280.log 2023-11-24T09:15:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77788.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77788.log.gz 2023-11-24T09:15:32.608 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44265.log 2023-11-24T09:15:32.609 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56280.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56280.log.gz 2023-11-24T09:15:32.610 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56863.log 2023-11-24T09:15:32.611 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.44265.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.44265.log.gz 2023-11-24T09:15:32.611 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51957.log 2023-11-24T09:15:32.612 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56863.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56863.log.gz 2023-11-24T09:15:32.613 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65549.log 2023-11-24T09:15:32.626 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51957.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.66058.log 2023-11-24T09:15:32.627 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65549.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65549.log.gz 2023-11-24T09:15:32.646 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46787.log 2023-11-24T09:15:32.654 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.66058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59453.log 2023-11-24T09:15:32.667 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46787.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61844.log 2023-11-24T09:15:32.667 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.46787.log.gz 2023-11-24T09:15:32.668 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59453.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59453.log.gz 2023-11-24T09:15:32.674 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67650.log 2023-11-24T09:15:32.676 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61844.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61844.log.gz 2023-11-24T09:15:32.678 INFO:teuthology.orchestra.run.smithi164.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.66058.log.gz 2023-11-24T09:15:32.694 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39559.log 2023-11-24T09:15:32.695 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67650.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67650.log.gz 2023-11-24T09:15:32.711 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75621.log 2023-11-24T09:15:32.711 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.39559.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39559.log.gz 2023-11-24T09:15:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76578.log 2023-11-24T09:15:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75621.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75621.log.gz 2023-11-24T09:15:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73947.log 2023-11-24T09:15:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76578.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76578.log.gz 2023-11-24T09:15:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65887.log 2023-11-24T09:15:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73947.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73947.log.gz 2023-11-24T09:15:32.746 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43945.log 2023-11-24T09:15:32.748 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65887.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65887.log.gz 2023-11-24T09:15:32.754 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68319.log 2023-11-24T09:15:32.755 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43945.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43945.log.gz 2023-11-24T09:15:32.758 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40786.log.gz 2023-11-24T09:15:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34685.log 2023-11-24T09:15:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68319.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68319.log.gz 2023-11-24T09:15:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39609.log 2023-11-24T09:15:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.34685.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34685.log.gz 2023-11-24T09:15:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60079.log 2023-11-24T09:15:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.39609.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39609.log.gz 2023-11-24T09:15:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77315.log 2023-11-24T09:15:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60079.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60079.log.gz 2023-11-24T09:15:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74068.log 2023-11-24T09:15:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.77315.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77315.log.gz 2023-11-24T09:15:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78438.log 2023-11-24T09:15:32.774 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54104.log 2023-11-24T09:15:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78438.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78438.log.gz 2023-11-24T09:15:32.780 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53216.log.gz 2023-11-24T09:15:32.788 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51746.log.gz 2023-11-24T09:15:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43922.log 2023-11-24T09:15:32.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2023-11-24T09:15:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39333.log 2023-11-24T09:15:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54104.log: /var/log/ceph/cluster2-client.mirror.43922.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43922.log.gz 2023-11-24T09:15:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68402.log 2023-11-24T09:15:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39333.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39333.log.gz 2023-11-24T09:15:32.794 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54104.log.gz 2023-11-24T09:15:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43458.log 2023-11-24T09:15:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52104.log 2023-11-24T09:15:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68402.log: /var/log/ceph/cluster2-client.mirror.43458.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68402.log.gz 2023-11-24T09:15:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58156.log 2023-11-24T09:15:32.795 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43458.log.gz 2023-11-24T09:15:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52104.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52104.log.gz 2023-11-24T09:15:32.797 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56059.log 2023-11-24T09:15:32.797 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46509.log 2023-11-24T09:15:32.797 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58156.log: /var/log/ceph/cluster2-client.mirror.56059.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58156.log.gz 2023-11-24T09:15:32.797 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59344.log 2023-11-24T09:15:32.798 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56059.log.gz 2023-11-24T09:15:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46509.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46509.log.gz 2023-11-24T09:15:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54935.log 2023-11-24T09:15:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45750.log 2023-11-24T09:15:32.804 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59344.log: /var/log/ceph/cluster2-client.mirror.54935.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59344.log.gz 2023-11-24T09:15:32.805 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54935.log.gz 2023-11-24T09:15:32.805 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51645.log 2023-11-24T09:15:32.805 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74888.log 2023-11-24T09:15:32.807 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.45750.log: /var/log/ceph/cluster2-client.mirror.51645.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45750.log.gz 2023-11-24T09:15:32.807 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51645.log.gz 2023-11-24T09:15:32.808 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78308.log 2023-11-24T09:15:32.808 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45414.log 2023-11-24T09:15:32.808 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74888.log: /var/log/ceph/cluster2-client.admin.78308.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78308.log.gz 2023-11-24T09:15:32.808 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74888.log.gz 2023-11-24T09:15:32.808 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62520.log 2023-11-24T09:15:32.808 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54329.log 2023-11-24T09:15:32.810 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45414.log: /var/log/ceph/cluster1-client.mirror.62520.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45414.log.gz 2023-11-24T09:15:32.810 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62520.log.gz 2023-11-24T09:15:32.810 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78204.log 2023-11-24T09:15:32.810 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64331.log 2023-11-24T09:15:32.810 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54329.log: /var/log/ceph/cluster2-client.admin.78204.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78204.log.gz 2023-11-24T09:15:32.811 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54329.log.gz 2023-11-24T09:15:32.811 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47545.log 2023-11-24T09:15:32.812 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79660.log 2023-11-24T09:15:32.812 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64331.log: /var/log/ceph/cluster2-client.mirror.47545.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47545.log.gz 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64331.log.gz 2023-11-24T09:15:32.812 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:32.813 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74673.log 2023-11-24T09:15:32.815 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74871.log 2023-11-24T09:15:32.815 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79660.log: /var/log/ceph/cluster1-client.mirror.74673.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79660.log.gz 2023-11-24T09:15:32.815 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61754.log 2023-11-24T09:15:32.815 INFO:teuthology.orchestra.run.smithi164.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74673.log.gz 2023-11-24T09:15:32.816 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74871.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74871.log.gz 2023-11-24T09:15:32.816 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.47330.log 2023-11-24T09:15:32.816 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54764.log 2023-11-24T09:15:32.816 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61754.log: /var/log/ceph/cluster1-client.mirror.0.47330.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61754.log.gz 2023-11-24T09:15:32.817 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44606.log 2023-11-24T09:15:32.817 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54764.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54764.log.gz 2023-11-24T09:15:32.817 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50212.log 2023-11-24T09:15:32.818 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44606.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44606.log.gz 2023-11-24T09:15:32.819 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73838.log 2023-11-24T09:15:32.819 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50212.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50212.log.gz 2023-11-24T09:15:32.821 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70225.log 2023-11-24T09:15:32.821 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.73838.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73838.log.gz 2023-11-24T09:15:32.822 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46615.log 2023-11-24T09:15:32.822 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70225.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70225.log.gz 2023-11-24T09:15:32.822 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51450.log 2023-11-24T09:15:32.828 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46615.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46615.log.gz 2023-11-24T09:15:32.828 INFO:teuthology.orchestra.run.smithi164.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.47330.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61439.log 2023-11-24T09:15:32.828 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:32.828 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51450.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51450.log.gz 2023-11-24T09:15:32.829 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69526.log 2023-11-24T09:15:32.829 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34837.log 2023-11-24T09:15:32.829 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61439.log: /var/log/ceph/cluster2-client.mirror.69526.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61439.log.gz 2023-11-24T09:15:32.830 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46853.log 2023-11-24T09:15:32.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.34837.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34837.log.gz 2023-11-24T09:15:32.831 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55361.log 2023-11-24T09:15:32.831 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69526.log.gz 2023-11-24T09:15:32.836 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46853.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46853.log.gz 2023-11-24T09:15:32.837 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59123.log 2023-11-24T09:15:32.837 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61529.log 2023-11-24T09:15:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55361.log: /var/log/ceph/cluster1-client.mirror.59123.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59123.log.gz 2023-11-24T09:15:32.839 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55361.log.gz 2023-11-24T09:15:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63787.log 2023-11-24T09:15:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56551.log 2023-11-24T09:15:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61529.log: /var/log/ceph/cluster1-client.mirror.63787.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61529.log.gz 2023-11-24T09:15:32.840 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65463.log 2023-11-24T09:15:32.841 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56551.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63787.log.gz 2023-11-24T09:15:32.841 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56551.log.gz 2023-11-24T09:15:32.842 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40419.log 2023-11-24T09:15:32.843 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60704.log 2023-11-24T09:15:32.843 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40419.log: /var/log/ceph/cluster2-client.mirror.65463.log: 55.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40419.log.gz 2023-11-24T09:15:32.845 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65463.log.gz 2023-11-24T09:15:32.845 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51957.log.gz 2023-11-24T09:15:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40231.log 2023-11-24T09:15:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.46981.log 2023-11-24T09:15:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60704.log: /var/log/ceph/cluster1-client.mirror.40231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71058.log 2023-11-24T09:15:32.845 INFO:teuthology.orchestra.run.smithi164.stderr: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40231.log.gz 2023-11-24T09:15:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.46981.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60704.log.gz 2023-11-24T09:15:32.850 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61777.log 2023-11-24T09:15:32.850 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66570.log 2023-11-24T09:15:32.850 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71058.log: /var/log/ceph/cluster1-client.mirror.61777.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71058.log.gz 2023-11-24T09:15:32.850 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61777.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58087.log 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.46981.log.gz 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66570.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66570.log.gz 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41272.log 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66193.log 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58087.log: /var/log/ceph/cluster1-client.admin.41272.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44165.log 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66193.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58087.log.gz 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41272.log.gz 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67526.log 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66193.log.gz 2023-11-24T09:15:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46044.log 2023-11-24T09:15:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44165.log: /var/log/ceph/cluster2-client.mirror.67526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48773.log 2023-11-24T09:15:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46044.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67526.log.gz 2023-11-24T09:15:32.852 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46044.log.gz 2023-11-24T09:15:32.858 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66147.log 2023-11-24T09:15:32.859 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73433.log 2023-11-24T09:15:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48773.log: /var/log/ceph/cluster1-client.mirror.66147.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66147.log.gz 2023-11-24T09:15:32.861 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48773.log.gz 2023-11-24T09:15:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49405.log 2023-11-24T09:15:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60967.log 2023-11-24T09:15:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.73433.log: /var/log/ceph/cluster2-client.mirror.49405.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73433.log.gz 2023-11-24T09:15:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70495.log 2023-11-24T09:15:32.862 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6%/var/log/ceph/cluster2-client.mirror.60967.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49405.log.gz 2023-11-24T09:15:32.863 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60967.log.gz 2023-11-24T09:15:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67312.log 2023-11-24T09:15:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73411.log 2023-11-24T09:15:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70495.log: /var/log/ceph/cluster2-client.mirror.67312.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70495.log.gz 2023-11-24T09:15:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55275.log 2023-11-24T09:15:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73411.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73411.log.gz 2023-11-24T09:15:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52850.log 2023-11-24T09:15:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55275.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55275.log.gz 2023-11-24T09:15:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46681.log 2023-11-24T09:15:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52850.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51509.log.gz 2023-11-24T09:15:32.868 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52850.log.gz 2023-11-24T09:15:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.55855.log 2023-11-24T09:15:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66391.log 2023-11-24T09:15:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.55855.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67312.log.gz 2023-11-24T09:15:32.874 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46681.log.gz 2023-11-24T09:15:32.883 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67753.log 2023-11-24T09:15:32.883 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77762.log 2023-11-24T09:15:32.884 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.67753.log: -- replaced with /var/log/ceph/cluster1-client.mirror.66391.log.gz 2023-11-24T09:15:32.884 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54239.log 2023-11-24T09:15:32.885 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77762.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77762.log.gz 2023-11-24T09:15:32.885 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.73744.log 2023-11-24T09:15:32.886 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54239.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54239.log.gz 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67753.log.gz 2023-11-24T09:15:32.886 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:32.887 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74719.log 2023-11-24T09:15:32.887 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.73744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44101.log 2023-11-24T09:15:32.889 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74719.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50706.log.gz 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74719.log.gz 2023-11-24T09:15:32.890 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:32.890 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79504.log 2023-11-24T09:15:32.890 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74026.log 2023-11-24T09:15:32.890 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44101.log: /var/log/ceph/cluster2-client.admin.79504.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79504.log.gz 2023-11-24T09:15:32.890 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65866.log 2023-11-24T09:15:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74026.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74026.log.gz 2023-11-24T09:15:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51872.log 2023-11-24T09:15:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65866.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65866.log.gz 2023-11-24T09:15:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70833.log 2023-11-24T09:15:32.896 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51872.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51872.log.gz 2023-11-24T09:15:32.896 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69196.log 2023-11-24T09:15:32.897 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44101.log.gz 2023-11-24T09:15:32.897 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42170.log 2023-11-24T09:15:32.897 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69196.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70833.log.gz 2023-11-24T09:15:32.897 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69196.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.51809.log 2023-11-24T09:15:32.897 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:32.897 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64180.log 2023-11-24T09:15:32.897 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42170.log: /var/log/ceph/cluster2-client.mirror.3.51809.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42170.log.gz 2023-11-24T09:15:32.897 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47745.log 2023-11-24T09:15:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64180.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64180.log.gz 2023-11-24T09:15:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44628.log 2023-11-24T09:15:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47745.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47745.log.gz 2023-11-24T09:15:32.901 INFO:teuthology.orchestra.run.smithi164.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.73744.log.gz 2023-11-24T09:15:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77866.log 2023-11-24T09:15:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43721.log 2023-11-24T09:15:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44628.log: /var/log/ceph/cluster2-client.admin.77866.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77866.log.gz 2023-11-24T09:15:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52892.log 2023-11-24T09:15:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43721.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.55855.log.gz 2023-11-24T09:15:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64351.log 2023-11-24T09:15:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52892.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.3.51809.log.gz 2023-11-24T09:15:32.912 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74009.log 2023-11-24T09:15:32.914 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64351.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64351.log.gz 2023-11-24T09:15:32.914 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.76852.log 2023-11-24T09:15:32.914 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74009.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74009.log.gz 2023-11-24T09:15:32.915 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64420.log 2023-11-24T09:15:32.926 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.76852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78776.log 2023-11-24T09:15:32.927 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64420.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64420.log.gz 2023-11-24T09:15:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50481.log 2023-11-24T09:15:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78776.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78776.log.gz 2023-11-24T09:15:32.959 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75355.log 2023-11-24T09:15:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50481.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50481.log.gz 2023-11-24T09:15:32.967 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46355.log 2023-11-24T09:15:32.967 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75355.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75355.log.gz 2023-11-24T09:15:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50292.log 2023-11-24T09:15:32.975 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46355.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46355.log.gz 2023-11-24T09:15:32.983 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75095.log 2023-11-24T09:15:32.983 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50292.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50292.log.gz 2023-11-24T09:15:32.990 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65641.log 2023-11-24T09:15:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75095.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75095.log.gz 2023-11-24T09:15:32.996 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74068.log.gz 2023-11-24T09:15:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65441.log 2023-11-24T09:15:32.997 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65641.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65641.log.gz 2023-11-24T09:15:32.997 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70810.log 2023-11-24T09:15:32.999 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65441.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65441.log.gz 2023-11-24T09:15:33.000 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78672.log 2023-11-24T09:15:33.000 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70810.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70810.log.gz 2023-11-24T09:15:33.001 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68691.log 2023-11-24T09:15:33.001 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78672.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78672.log.gz 2023-11-24T09:15:33.002 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76761.log 2023-11-24T09:15:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68691.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68691.log.gz 2023-11-24T09:15:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60171.log 2023-11-24T09:15:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76761.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76761.log.gz 2023-11-24T09:15:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70405.log 2023-11-24T09:15:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60171.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60171.log.gz 2023-11-24T09:15:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62261.log 2023-11-24T09:15:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70405.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70405.log.gz 2023-11-24T09:15:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74965.log 2023-11-24T09:15:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62261.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62261.log.gz 2023-11-24T09:15:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50627.log 2023-11-24T09:15:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74965.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74965.log.gz 2023-11-24T09:15:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57039.log 2023-11-24T09:15:33.019 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45070.log 2023-11-24T09:15:33.034 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57039.log: 92.1% 95.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.76852.log.gz 2023-11-24T09:15:33.035 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57039.log.gz 2023-11-24T09:15:33.035 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.68899.log 2023-11-24T09:15:33.035 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45070.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45070.log.gz 2023-11-24T09:15:33.036 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62606.log 2023-11-24T09:15:33.041 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.68899.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.68899.log.gz 2023-11-24T09:15:33.042 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58903.log 2023-11-24T09:15:33.044 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62606.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62606.log.gz 2023-11-24T09:15:33.044 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76058.log 2023-11-24T09:15:33.045 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58903.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58903.log.gz 2023-11-24T09:15:33.045 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46133.log 2023-11-24T09:15:33.047 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76058.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76058.log.gz 2023-11-24T09:15:33.047 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77996.log 2023-11-24T09:15:33.048 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46133.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46133.log.gz 2023-11-24T09:15:33.049 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57672.log 2023-11-24T09:15:33.049 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77996.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77996.log.gz 2023-11-24T09:15:33.050 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40679.log 2023-11-24T09:15:33.051 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57672.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57672.log.gz 2023-11-24T09:15:33.051 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67146.log 2023-11-24T09:15:33.065 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40679.log: 94.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72070.log 2023-11-24T09:15:33.067 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67146.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67146.log.gz 2023-11-24T09:15:33.067 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40679.log.gz 2023-11-24T09:15:33.068 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77502.log 2023-11-24T09:15:33.068 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72070.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72070.log.gz 2023-11-24T09:15:33.068 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46179.log 2023-11-24T09:15:33.069 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77502.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77502.log.gz 2023-11-24T09:15:33.069 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77409.log 2023-11-24T09:15:33.070 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46179.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46179.log.gz 2023-11-24T09:15:33.071 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39177.log 2023-11-24T09:15:33.071 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.77409.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77409.log.gz 2023-11-24T09:15:33.072 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48677.log 2023-11-24T09:15:33.072 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39177.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39177.log.gz 2023-11-24T09:15:33.072 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55399.log 2023-11-24T09:15:33.073 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48677.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48677.log.gz 2023-11-24T09:15:33.073 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73666.log 2023-11-24T09:15:33.075 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55399.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55399.log.gz 2023-11-24T09:15:33.075 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76740.log 2023-11-24T09:15:33.076 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73666.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73666.log.gz 2023-11-24T09:15:33.076 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65041.log 2023-11-24T09:15:33.077 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76740.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76740.log.gz 2023-11-24T09:15:33.078 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66256.log 2023-11-24T09:15:33.078 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65041.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65041.log.gz 2023-11-24T09:15:33.079 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50822.log 2023-11-24T09:15:33.080 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66256.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66256.log.gz 2023-11-24T09:15:33.080 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79894.log 2023-11-24T09:15:33.082 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50822.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50822.log.gz 2023-11-24T09:15:33.082 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68278.log 2023-11-24T09:15:33.083 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79894.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79894.log.gz 2023-11-24T09:15:33.083 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65682.log 2023-11-24T09:15:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68278.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68278.log.gz 2023-11-24T09:15:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49994.log 2023-11-24T09:15:33.087 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65682.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65682.log.gz 2023-11-24T09:15:33.087 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55575.log 2023-11-24T09:15:33.088 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49994.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49994.log.gz 2023-11-24T09:15:33.088 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71103.log 2023-11-24T09:15:33.091 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55575.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55575.log.gz 2023-11-24T09:15:33.091 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41790.log 2023-11-24T09:15:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71103.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71103.log.gz 2023-11-24T09:15:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71485.log 2023-11-24T09:15:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41790.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41790.log.gz 2023-11-24T09:15:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41667.log 2023-11-24T09:15:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71485.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71485.log.gz 2023-11-24T09:15:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49277.log 2023-11-24T09:15:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41667.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41667.log.gz 2023-11-24T09:15:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47265.log 2023-11-24T09:15:33.118 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53687.log 2023-11-24T09:15:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47265.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43721.log.gz 2023-11-24T09:15:33.125 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47265.log.gz 2023-11-24T09:15:33.125 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67443.log 2023-11-24T09:15:33.126 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53687.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53687.log.gz 2023-11-24T09:15:33.126 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45437.log 2023-11-24T09:15:33.129 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67443.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67443.log.gz 2023-11-24T09:15:33.129 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73546.log 2023-11-24T09:15:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45437.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45437.log.gz 2023-11-24T09:15:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53481.log 2023-11-24T09:15:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.73546.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73546.log.gz 2023-11-24T09:15:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61079.log 2023-11-24T09:15:33.146 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73390.log 2023-11-24T09:15:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61079.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61079.log.gz 2023-11-24T09:15:33.154 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78802.log 2023-11-24T09:15:33.155 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.73390.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73390.log.gz 2023-11-24T09:15:33.170 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79218.log 2023-11-24T09:15:33.171 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78802.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78802.log.gz 2023-11-24T09:15:33.178 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64812.log 2023-11-24T09:15:33.179 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79218.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79218.log.gz 2023-11-24T09:15:33.187 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76598.log 2023-11-24T09:15:33.188 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64812.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64812.log.gz 2023-11-24T09:15:33.190 INFO:teuthology.orchestra.run.smithi164.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53481.log.gz 2023-11-24T09:15:33.191 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61338.log 2023-11-24T09:15:33.193 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76598.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76598.log.gz 2023-11-24T09:15:33.193 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42712.log 2023-11-24T09:15:33.194 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61338.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61338.log.gz 2023-11-24T09:15:33.195 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39508.log 2023-11-24T09:15:33.196 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42712.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42712.log.gz 2023-11-24T09:15:33.196 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78750.log 2023-11-24T09:15:33.197 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39508.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39508.log.gz 2023-11-24T09:15:33.197 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67609.log 2023-11-24T09:15:33.198 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78750.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78750.log.gz 2023-11-24T09:15:33.198 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57603.log 2023-11-24T09:15:33.199 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67609.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67609.log.gz 2023-11-24T09:15:33.199 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44650.log 2023-11-24T09:15:33.200 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57603.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57603.log.gz 2023-11-24T09:15:33.201 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68609.log 2023-11-24T09:15:33.206 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44650.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44650.log.gz 2023-11-24T09:15:33.206 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65118.log 2023-11-24T09:15:33.207 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68609.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68609.log.gz 2023-11-24T09:15:33.208 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51566.log 2023-11-24T09:15:33.209 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65118.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65118.log.gz 2023-11-24T09:15:33.209 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48868.log 2023-11-24T09:15:33.210 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51566.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51566.log.gz 2023-11-24T09:15:33.210 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68795.log 2023-11-24T09:15:33.211 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48868.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48868.log.gz 2023-11-24T09:15:33.211 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47330.log 2023-11-24T09:15:33.212 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9%/var/log/ceph/cluster2-client.mirror.68795.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52892.log.gz 2023-11-24T09:15:33.212 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68795.log.gz 2023-11-24T09:15:33.213 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54196.log 2023-11-24T09:15:33.213 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55855.log 2023-11-24T09:15:33.214 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47330.log: /var/log/ceph/cluster2-client.mirror.54196.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54196.log.gz 2023-11-24T09:15:33.214 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47330.log.gz 2023-11-24T09:15:33.215 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58590.log 2023-11-24T09:15:33.215 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71800.log 2023-11-24T09:15:33.216 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55855.log: /var/log/ceph/cluster1-client.mirror.58590.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58590.log.gz 2023-11-24T09:15:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49785.log 2023-11-24T09:15:33.227 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71800.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71800.log.gz 2023-11-24T09:15:33.227 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57169.log 2023-11-24T09:15:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49785.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55855.log.gz 2023-11-24T09:15:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78360.log 2023-11-24T09:15:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57169.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57169.log.gz 2023-11-24T09:15:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48985.log 2023-11-24T09:15:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78360.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78360.log.gz 2023-11-24T09:15:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53396.log 2023-11-24T09:15:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48985.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48985.log.gz 2023-11-24T09:15:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69650.log 2023-11-24T09:15:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53396.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53396.log.gz 2023-11-24T09:15:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.52603.log 2023-11-24T09:15:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69650.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69650.log.gz 2023-11-24T09:15:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48015.log 2023-11-24T09:15:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.3.52603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41580.log 2023-11-24T09:15:33.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48015.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44628.log.gz 2023-11-24T09:15:33.278 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49277.log.gz 2023-11-24T09:15:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63369.log 2023-11-24T09:15:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41580.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41580.log.gz 2023-11-24T09:15:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78568.log 2023-11-24T09:15:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63369.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63369.log.gz 2023-11-24T09:15:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43832.log 2023-11-24T09:15:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78568.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78568.log.gz 2023-11-24T09:15:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63563.log 2023-11-24T09:15:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43832.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43832.log.gz 2023-11-24T09:15:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64952.log 2023-11-24T09:15:33.284 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63563.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63563.log.gz 2023-11-24T09:15:33.284 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77146.log 2023-11-24T09:15:33.286 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64952.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49785.log.gz 2023-11-24T09:15:33.286 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73608.log 2023-11-24T09:15:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.77146.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77146.log.gz 2023-11-24T09:15:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68567.log 2023-11-24T09:15:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73608.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73608.log.gz 2023-11-24T09:15:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45111.log 2023-11-24T09:15:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68567.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68567.log.gz 2023-11-24T09:15:33.294 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78594.log 2023-11-24T09:15:33.299 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45111.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45111.log.gz 2023-11-24T09:15:33.300 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42280.log 2023-11-24T09:15:33.301 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78594.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78594.log.gz 2023-11-24T09:15:33.301 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57339.log 2023-11-24T09:15:33.301 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64952.log.gz 2023-11-24T09:15:33.302 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42796.log 2023-11-24T09:15:33.303 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57339.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57339.log.gz 2023-11-24T09:15:33.303 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44165.log.gz 2023-11-24T09:15:33.303 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71328.log 2023-11-24T09:15:33.305 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63890.log 2023-11-24T09:15:33.305 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42796.log: /var/log/ceph/cluster1-client.mirror.71328.log: 87.4% 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71328.log.gz 2023-11-24T09:15:33.305 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42796.log.gz 2023-11-24T09:15:33.305 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45301.log 2023-11-24T09:15:33.305 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67919.log 2023-11-24T09:15:33.307 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45301.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63890.log.gz 2023-11-24T09:15:33.307 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61552.log 2023-11-24T09:15:33.307 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45301.log.gz 2023-11-24T09:15:33.307 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42369.log 2023-11-24T09:15:33.307 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42280.log.gz 2023-11-24T09:15:33.308 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61552.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48366.log 2023-11-24T09:15:33.308 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67919.log.gz 2023-11-24T09:15:33.308 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61552.log.gz 2023-11-24T09:15:33.308 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59770.log 2023-11-24T09:15:33.311 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.42369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62238.log 2023-11-24T09:15:33.312 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.42369.log.gz 2023-11-24T09:15:33.312 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48366.log: /var/log/ceph/cluster1-client.mirror.59770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67774.log 2023-11-24T09:15:33.312 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62238.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62238.log.gz 2023-11-24T09:15:33.312 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58768.log 2023-11-24T09:15:33.312 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59770.log.gz 2023-11-24T09:15:33.312 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67774.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67774.log.gz 2023-11-24T09:15:33.312 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42583.log 2023-11-24T09:15:33.313 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58768.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69402.log 2023-11-24T09:15:33.314 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42583.log: 88.1% 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58768.log.gz 2023-11-24T09:15:33.315 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42583.log.gz 2023-11-24T09:15:33.315 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72359.log 2023-11-24T09:15:33.315 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53315.log 2023-11-24T09:15:33.315 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.72359.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72359.log.gz 2023-11-24T09:15:33.315 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59858.log 2023-11-24T09:15:33.315 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53315.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53315.log.gz 2023-11-24T09:15:33.316 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70338.log 2023-11-24T09:15:33.316 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69402.log.gz 2023-11-24T09:15:33.316 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58202.log 2023-11-24T09:15:33.316 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59858.log: 87.5%/var/log/ceph/cluster1-client.mirror.70338.log: -- replaced with /var/log/ceph/cluster2-client.mirror.59858.log.gz 2023-11-24T09:15:33.316 INFO:teuthology.orchestra.run.smithi164.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70338.log.gz 2023-11-24T09:15:33.318 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61505.log 2023-11-24T09:15:33.319 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74328.log 2023-11-24T09:15:33.319 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61505.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58202.log.gz 2023-11-24T09:15:33.319 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61505.log.gz 2023-11-24T09:15:33.319 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47805.log 2023-11-24T09:15:33.319 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51172.log 2023-11-24T09:15:33.319 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.47805.log: -- replaced with /var/log/ceph/cluster2-client.mirror.74328.log.gz 2023-11-24T09:15:33.320 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47805.log.gz 2023-11-24T09:15:33.320 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49741.log 2023-11-24T09:15:33.320 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44295.log 2023-11-24T09:15:33.321 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49741.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49741.log.gz 2023-11-24T09:15:33.321 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72318.log 2023-11-24T09:15:33.321 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44295.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44295.log.gz 2023-11-24T09:15:33.321 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69692.log 2023-11-24T09:15:33.322 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72318.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72318.log.gz 2023-11-24T09:15:33.323 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44935.log 2023-11-24T09:15:33.323 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69692.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69692.log.gz 2023-11-24T09:15:33.323 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60595.log 2023-11-24T09:15:33.324 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44935.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44935.log.gz 2023-11-24T09:15:33.325 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42147.log 2023-11-24T09:15:33.325 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60595.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60595.log.gz 2023-11-24T09:15:33.326 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.39813.log 2023-11-24T09:15:33.326 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42147.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42147.log.gz 2023-11-24T09:15:33.327 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69258.log 2023-11-24T09:15:33.327 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.39813.log: 76.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.39813.log.gz 2023-11-24T09:15:33.328 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68547.log 2023-11-24T09:15:33.328 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69258.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50627.log.gz 2023-11-24T09:15:33.328 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69258.log.gz 2023-11-24T09:15:33.329 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68009.log 2023-11-24T09:15:33.330 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49126.log 2023-11-24T09:15:33.330 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68009.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68547.log.gz 2023-11-24T09:15:33.332 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68009.log.gz 2023-11-24T09:15:33.332 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70248.log 2023-11-24T09:15:33.334 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41511.log 2023-11-24T09:15:33.334 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49126.log: /var/log/ceph/cluster1-client.mirror.70248.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49126.log.gz 2023-11-24T09:15:33.334 INFO:teuthology.orchestra.run.smithi164.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70248.log.gz 2023-11-24T09:15:33.334 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40265.log 2023-11-24T09:15:33.334 INFO:teuthology.orchestra.run.smithi164.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51172.log.gz 2023-11-24T09:15:33.334 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41488.log 2023-11-24T09:15:33.336 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41350.log 2023-11-24T09:15:33.336 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41511.log.gz 2023-11-24T09:15:33.336 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.40265.log: /var/log/ceph/cluster2-client.mirror.41488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46744.log 2023-11-24T09:15:33.336 INFO:teuthology.orchestra.run.smithi164.stderr: 58.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40265.log.gz 2023-11-24T09:15:33.336 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41488.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65618.log 2023-11-24T09:15:33.336 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:33.336 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.41350.log: /var/log/ceph/cluster1-client.mirror.46744.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.41350.log.gz 2023-11-24T09:15:33.337 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63700.log 2023-11-24T09:15:33.337 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53819.log 2023-11-24T09:15:33.338 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63700.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65618.log.gz 2023-11-24T09:15:33.338 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63700.log.gz 2023-11-24T09:15:33.338 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.50373.log 2023-11-24T09:15:33.338 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53819.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53819.log.gz 2023-11-24T09:15:33.339 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.48578.log 2023-11-24T09:15:33.339 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.3.50373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68443.log 2023-11-24T09:15:33.341 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.48578.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46744.log.gz 2023-11-24T09:15:33.342 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75877.log 2023-11-24T09:15:33.344 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68443.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68443.log.gz 2023-11-24T09:15:33.345 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62687.log 2023-11-24T09:15:33.346 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75877.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75877.log.gz 2023-11-24T09:15:33.346 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52950.log 2023-11-24T09:15:33.347 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62687.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62687.log.gz 2023-11-24T09:15:33.347 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76942.log 2023-11-24T09:15:33.348 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52950.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52950.log.gz 2023-11-24T09:15:33.348 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58811.log 2023-11-24T09:15:33.349 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76942.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76942.log.gz 2023-11-24T09:15:33.350 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49704.log 2023-11-24T09:15:33.350 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58811.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58811.log.gz 2023-11-24T09:15:33.351 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71373.log 2023-11-24T09:15:33.351 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49704.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49704.log.gz 2023-11-24T09:15:33.352 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.39818.log 2023-11-24T09:15:33.353 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71373.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71373.log.gz 2023-11-24T09:15:33.353 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60194.log 2023-11-24T09:15:33.358 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.39818.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.39818.log.gz 2023-11-24T09:15:33.359 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76513.log 2023-11-24T09:15:33.360 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60194.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60194.log.gz 2023-11-24T09:15:33.360 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71913.log 2023-11-24T09:15:33.361 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76513.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76513.log.gz 2023-11-24T09:15:33.361 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42399.log 2023-11-24T09:15:33.362 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71913.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71913.log.gz 2023-11-24T09:15:33.362 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50271.log 2023-11-24T09:15:33.363 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42399.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42399.log.gz 2023-11-24T09:15:33.364 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70675.log 2023-11-24T09:15:33.365 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50271.log: 92.6% 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.3.50373.log.gz 2023-11-24T09:15:33.365 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.3.48578.log.gz 2023-11-24T09:15:33.365 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53174.log 2023-11-24T09:15:33.366 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68989.log 2023-11-24T09:15:33.366 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53174.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70675.log.gz 2023-11-24T09:15:33.366 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79452.log 2023-11-24T09:15:33.367 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68989.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53174.log.gz 2023-11-24T09:15:33.367 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68989.log.gz 2023-11-24T09:15:33.368 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70022.log 2023-11-24T09:15:33.368 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53923.log 2023-11-24T09:15:33.368 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79452.log.gz 2023-11-24T09:15:33.369 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.70022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52522.log 2023-11-24T09:15:33.369 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53923.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53923.log.gz 2023-11-24T09:15:33.370 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68671.log 2023-11-24T09:15:33.370 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52522.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52522.log.gz 2023-11-24T09:15:33.370 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 90.4% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.70022.log.gz 2023-11-24T09:15:33.370 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/cluster2-client.mirror.67485.log 2023-11-24T09:15:33.371 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68671.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68671.log.gz 2023-11-24T09:15:33.372 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56531.log 2023-11-24T09:15:33.372 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52025.log 2023-11-24T09:15:33.372 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56531.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67485.log.gz 2023-11-24T09:15:33.373 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42627.log 2023-11-24T09:15:33.373 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52025.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52025.log.gz 2023-11-24T09:15:33.373 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44978.log 2023-11-24T09:15:33.373 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56531.log.gz 2023-11-24T09:15:33.379 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42627.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42627.log.gz 2023-11-24T09:15:33.379 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67691.log 2023-11-24T09:15:33.380 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44978.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45371.log 2023-11-24T09:15:33.380 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67691.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44978.log.gz 2023-11-24T09:15:33.381 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60618.log 2023-11-24T09:15:33.382 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45371.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45371.log.gz 2023-11-24T09:15:33.382 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67691.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41534.log 2023-11-24T09:15:33.382 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:33.383 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60618.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60618.log.gz 2023-11-24T09:15:33.383 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69423.log 2023-11-24T09:15:33.383 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58722.log 2023-11-24T09:15:33.384 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69423.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41534.log.gz 2023-11-24T09:15:33.384 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69795.log 2023-11-24T09:15:33.385 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69423.log.gz 2023-11-24T09:15:33.385 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58722.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58722.log.gz 2023-11-24T09:15:33.386 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79062.log 2023-11-24T09:15:33.386 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.73433.log 2023-11-24T09:15:33.386 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79062.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79062.log.gz 2023-11-24T09:15:33.390 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69795.log.gz 2023-11-24T09:15:33.390 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49447.log 2023-11-24T09:15:33.390 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.73433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49484.log 2023-11-24T09:15:33.396 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49447.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.73433.log.gz 2023-11-24T09:15:33.396 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40282.log 2023-11-24T09:15:33.398 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49484.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49484.log.gz 2023-11-24T09:15:33.398 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70113.log 2023-11-24T09:15:33.399 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.40282.log: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40282.log.gz 2023-11-24T09:15:33.399 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69113.log 2023-11-24T09:15:33.400 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70113.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70113.log.gz 2023-11-24T09:15:33.400 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43343.log 2023-11-24T09:15:33.401 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69113.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69113.log.gz 2023-11-24T09:15:33.402 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57019.log 2023-11-24T09:15:33.402 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43343.log: 81.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43343.log.gz 2023-11-24T09:15:33.403 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45547.log 2023-11-24T09:15:33.404 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57019.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57019.log.gz 2023-11-24T09:15:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59209.log 2023-11-24T09:15:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45547.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45547.log.gz 2023-11-24T09:15:33.411 INFO:teuthology.orchestra.run.smithi164.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-11-24T09:15:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51529.log 2023-11-24T09:15:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77892.log 2023-11-24T09:15:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51529.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51529.log.gz 2023-11-24T09:15:33.412 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59209.log.gz 2023-11-24T09:15:33.414 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68092.log 2023-11-24T09:15:33.414 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79088.log 2023-11-24T09:15:33.414 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77892.log.gz 2023-11-24T09:15:33.414 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73744.log 2023-11-24T09:15:33.415 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79088.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79088.log.gz 89.5% 2023-11-24T09:15:33.415 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68092.log.gz 2023-11-24T09:15:33.415 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75770.log 2023-11-24T09:15:33.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44500.log 2023-11-24T09:15:33.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75770.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75770.log.gz 2023-11-24T09:15:33.416 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73744.log.gz 2023-11-24T09:15:33.416 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42344.log 2023-11-24T09:15:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67166.log 2023-11-24T09:15:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44500.log: /var/log/ceph/cluster2-client.mirror.42344.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44500.log.gz 2023-11-24T09:15:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57781.log 2023-11-24T09:15:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67166.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67166.log.gz 2023-11-24T09:15:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68154.log 2023-11-24T09:15:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57781.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57781.log.gz 2023-11-24T09:15:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75136.log 2023-11-24T09:15:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68154.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68154.log.gz 2023-11-24T09:15:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64599.log 2023-11-24T09:15:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75136.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75136.log.gz 2023-11-24T09:15:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39840.log 2023-11-24T09:15:33.425 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64599.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64599.log.gz 2023-11-24T09:15:33.425 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78386.log 2023-11-24T09:15:33.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39840.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48344.log 2023-11-24T09:15:33.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78386.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78386.log.gz 2023-11-24T09:15:33.443 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64091.log 2023-11-24T09:15:33.443 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48344.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48344.log.gz 2023-11-24T09:15:33.451 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39840.log.gz 2023-11-24T09:15:33.451 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60549.log 2023-11-24T09:15:33.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64091.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64091.log.gz 2023-11-24T09:15:33.453 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57359.log 2023-11-24T09:15:33.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60549.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60549.log.gz 2023-11-24T09:15:33.454 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46658.log 2023-11-24T09:15:33.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57359.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57359.log.gz 2023-11-24T09:15:33.455 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.59640.log 2023-11-24T09:15:33.461 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46658.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46658.log.gz 2023-11-24T09:15:33.461 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52539.log 2023-11-24T09:15:33.474 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.59640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54603.log 2023-11-24T09:15:33.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52539.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52539.log.gz 2023-11-24T09:15:33.489 INFO:teuthology.orchestra.run.smithi164.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.59640.log.gz 2023-11-24T09:15:33.489 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67464.log 2023-11-24T09:15:33.492 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54603.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54603.log.gz 2023-11-24T09:15:33.492 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59904.log 2023-11-24T09:15:33.493 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67464.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67464.log.gz 2023-11-24T09:15:33.494 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49185.log 2023-11-24T09:15:33.494 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59904.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59904.log.gz 2023-11-24T09:15:33.495 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68237.log 2023-11-24T09:15:33.506 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49185.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.52603.log.gz 2023-11-24T09:15:33.510 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65774.log 2023-11-24T09:15:33.512 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68237.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68237.log.gz 2023-11-24T09:15:33.512 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45047.log 2023-11-24T09:15:33.513 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65774.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65774.log.gz 2023-11-24T09:15:33.513 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40342.log 2023-11-24T09:15:33.514 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45047.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45047.log.gz 2023-11-24T09:15:33.514 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70158.log 2023-11-24T09:15:33.515 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.40342.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40342.log.gz 2023-11-24T09:15:33.516 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78412.log 2023-11-24T09:15:33.516 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70158.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70158.log.gz 2023-11-24T09:15:33.516 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41468.log 2023-11-24T09:15:33.517 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78412.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78412.log.gz 2023-11-24T09:15:33.517 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68051.log 2023-11-24T09:15:33.519 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41468.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41468.log.gz 2023-11-24T09:15:33.519 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50329.log 2023-11-24T09:15:33.520 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68051.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68051.log.gz 2023-11-24T09:15:33.520 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43110.log 2023-11-24T09:15:33.521 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50329.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50329.log.gz 2023-11-24T09:15:33.522 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49504.log 2023-11-24T09:15:33.523 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43110.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43110.log.gz 2023-11-24T09:15:33.523 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68899.log 2023-11-24T09:15:33.524 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49504.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48366.log.gz 2023-11-24T09:15:33.524 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49504.log.gz 2023-11-24T09:15:33.524 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63767.log 2023-11-24T09:15:33.525 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52717.log 2023-11-24T09:15:33.525 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.68899.log: /var/log/ceph/cluster2-client.mirror.63767.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68899.log.gz 2023-11-24T09:15:33.526 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66345.log 2023-11-24T09:15:33.527 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52717.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63767.log.gz 2023-11-24T09:15:33.527 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52717.log.gz 2023-11-24T09:15:33.527 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43025.log 2023-11-24T09:15:33.528 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58680.log 2023-11-24T09:15:33.528 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66345.log: /var/log/ceph/cluster2-client.mirror.43025.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66345.log.gz 2023-11-24T09:15:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79140.log 2023-11-24T09:15:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58680.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58680.log.gz 2023-11-24T09:15:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43699.log 2023-11-24T09:15:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79140.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79140.log.gz 2023-11-24T09:15:33.532 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68464.log 2023-11-24T09:15:33.533 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43699.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43699.log.gz 2023-11-24T09:15:33.533 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.76315.log 2023-11-24T09:15:33.534 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68464.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43025.log.gz 2023-11-24T09:15:33.534 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68464.log.gz 2023-11-24T09:15:33.534 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59838.log 2023-11-24T09:15:33.535 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40487.log 2023-11-24T09:15:33.536 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.76315.log: /var/log/ceph/cluster2-client.mirror.59838.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59838.log.gz 2023-11-24T09:15:33.537 INFO:teuthology.orchestra.run.smithi164.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.76315.log.gz 2023-11-24T09:15:33.538 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45483.log 2023-11-24T09:15:33.538 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67001.log 2023-11-24T09:15:33.539 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40487.log: /var/log/ceph/cluster1-client.mirror.45483.log: 58.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40487.log.gz 2023-11-24T09:15:33.539 INFO:teuthology.orchestra.run.smithi164.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41376.log 2023-11-24T09:15:33.539 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.45483.log.gz 2023-11-24T09:15:33.540 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67001.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67001.log.gz 2023-11-24T09:15:33.540 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65400.log 2023-11-24T09:15:33.541 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52972.log 2023-11-24T09:15:33.542 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.41376.log: /var/log/ceph/cluster1-client.mirror.65400.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.41376.log.gz 2023-11-24T09:15:33.542 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65400.log.gz 2023-11-24T09:15:33.542 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49911.log 2023-11-24T09:15:33.542 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54470.log 2023-11-24T09:15:33.543 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52972.log: /var/log/ceph/cluster2-client.mirror.49911.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49911.log.gz 2023-11-24T09:15:33.558 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75195.log 2023-11-24T09:15:33.559 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54470.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54470.log.gz 2023-11-24T09:15:33.560 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77736.log 2023-11-24T09:15:33.566 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61688.log 2023-11-24T09:15:33.575 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77736.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77736.log.gz 2023-11-24T09:15:33.587 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41689.log 2023-11-24T09:15:33.588 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61688.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61688.log.gz 2023-11-24T09:15:33.602 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44521.log 2023-11-24T09:15:33.604 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41689.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41689.log.gz 2023-11-24T09:15:33.610 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77970.log 2023-11-24T09:15:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44521.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51487.log 2023-11-24T09:15:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77970.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77970.log.gz 2023-11-24T09:15:33.631 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44521.log.gz 2023-11-24T09:15:33.631 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49447.log.gz 2023-11-24T09:15:33.631 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42081.log 2023-11-24T09:15:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51487.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51487.log.gz 2023-11-24T09:15:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67939.log 2023-11-24T09:15:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42081.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42081.log.gz 2023-11-24T09:15:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63540.log 2023-11-24T09:15:33.636 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67939.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67939.log.gz 2023-11-24T09:15:33.636 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46263.log 2023-11-24T09:15:33.637 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63540.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63540.log.gz 2023-11-24T09:15:33.637 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55071.log 2023-11-24T09:15:33.643 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46263.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46263.log.gz 2023-11-24T09:15:33.643 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69568.log 2023-11-24T09:15:33.645 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55071.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55071.log.gz 2023-11-24T09:15:33.645 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63982.log 2023-11-24T09:15:33.646 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69568.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69568.log.gz 2023-11-24T09:15:33.646 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76427.log 2023-11-24T09:15:33.647 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63982.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63982.log.gz 2023-11-24T09:15:33.647 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57451.log 2023-11-24T09:15:33.660 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76427.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42344.log.gz 2023-11-24T09:15:33.663 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79244.log 2023-11-24T09:15:33.663 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57451.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57451.log.gz 2023-11-24T09:15:33.664 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54718.log 2023-11-24T09:15:33.664 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79244.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79244.log.gz 2023-11-24T09:15:33.664 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65208.log 2023-11-24T09:15:33.665 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54718.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54718.log.gz 2023-11-24T09:15:33.665 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.49785.log 2023-11-24T09:15:33.667 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65208.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65208.log.gz 2023-11-24T09:15:33.668 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50140.log 2023-11-24T09:15:33.678 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.49785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47136.log 2023-11-24T09:15:33.679 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50140.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50140.log.gz 2023-11-24T09:15:33.686 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79712.log 2023-11-24T09:15:33.699 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47136.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.39895.log 2023-11-24T09:15:33.699 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79712.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79712.log.gz 2023-11-24T09:15:33.710 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.47136.log.gz 2023-11-24T09:15:33.714 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71148.log 2023-11-24T09:15:33.717 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.39895.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.39895.log.gz 2023-11-24T09:15:33.722 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79296.log 2023-11-24T09:15:33.723 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71148.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71148.log.gz 2023-11-24T09:15:33.738 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54873.log 2023-11-24T09:15:33.740 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79296.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79296.log.gz 2023-11-24T09:15:33.751 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50271.log.gz 2023-11-24T09:15:33.751 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44744.log 2023-11-24T09:15:33.752 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54873.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54873.log.gz 2023-11-24T09:15:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.64952.log 2023-11-24T09:15:33.754 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.44744.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.44744.log.gz 2023-11-24T09:15:33.754 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79010.log 2023-11-24T09:15:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.0.64952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77290.log 2023-11-24T09:15:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79010.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79010.log.gz 2023-11-24T09:15:33.776 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49185.log.gz 2023-11-24T09:15:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47866.log 2023-11-24T09:15:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.77290.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77290.log.gz 2023-11-24T09:15:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47308.log 2023-11-24T09:15:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47866.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47866.log.gz 2023-11-24T09:15:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70923.log 2023-11-24T09:15:33.785 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47308.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47308.log.gz 2023-11-24T09:15:33.786 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58133.log 2023-11-24T09:15:33.786 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70923.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70923.log.gz 2023-11-24T09:15:33.786 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66234.log 2023-11-24T09:15:33.787 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58133.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58133.log.gz 2023-11-24T09:15:33.787 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65331.log 2023-11-24T09:15:33.789 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66234.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66234.log.gz 2023-11-24T09:15:33.790 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46830.log 2023-11-24T09:15:33.791 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65331.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65331.log.gz 2023-11-24T09:15:33.791 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71755.log 2023-11-24T09:15:33.797 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46830.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46830.log.gz 2023-11-24T09:15:33.797 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69092.log 2023-11-24T09:15:33.798 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71755.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71755.log.gz 2023-11-24T09:15:33.798 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42322.log 2023-11-24T09:15:33.802 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69092.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69092.log.gz 2023-11-24T09:15:33.802 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61616.log 2023-11-24T09:15:33.802 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42322.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42322.log.gz 2023-11-24T09:15:33.802 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50805.log 2023-11-24T09:15:33.805 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61616.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61616.log.gz 2023-11-24T09:15:33.805 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78932.log 2023-11-24T09:15:33.805 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.64952.log.gz 2023-11-24T09:15:33.805 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50805.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50805.log.gz 2023-11-24T09:15:33.806 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58265.log 2023-11-24T09:15:33.807 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67733.log 2023-11-24T09:15:33.807 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78932.log.gz 2023-11-24T09:15:33.807 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56998.log 2023-11-24T09:15:33.809 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58265.log: /var/log/ceph/cluster2-client.mirror.67733.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67733.log.gz 2023-11-24T09:15:33.809 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58265.log.gz 2023-11-24T09:15:33.809 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48402.log 2023-11-24T09:15:33.809 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56998.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78880.log 2023-11-24T09:15:33.809 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56998.log.gz 2023-11-24T09:15:33.809 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48402.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48402.log.gz 2023-11-24T09:15:33.809 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79816.log 2023-11-24T09:15:33.812 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39709.log 2023-11-24T09:15:33.812 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78880.log: /var/log/ceph/cluster2-client.admin.79816.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78880.log.gz 2023-11-24T09:15:33.812 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79816.log.gz 2023-11-24T09:15:33.813 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62842.log 2023-11-24T09:15:33.813 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42606.log 2023-11-24T09:15:33.813 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39709.log: /var/log/ceph/cluster2-client.mirror.62842.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39709.log.gz 2023-11-24T09:15:33.813 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46064.log 2023-11-24T09:15:33.813 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62842.log.gz 2023-11-24T09:15:33.813 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42606.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42606.log.gz 2023-11-24T09:15:33.813 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58311.log 2023-11-24T09:15:33.815 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70743.log 2023-11-24T09:15:33.815 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46064.log: /var/log/ceph/cluster2-client.mirror.58311.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46064.log.gz 2023-11-24T09:15:33.815 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55091.log 2023-11-24T09:15:33.816 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58311.log.gz 2023-11-24T09:15:33.816 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70743.log: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70743.log.gz 2023-11-24T09:15:33.816 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48815.log 2023-11-24T09:15:33.816 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56196.log 2023-11-24T09:15:33.817 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55091.log: /var/log/ceph/cluster2-client.mirror.48815.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55091.log.gz 2023-11-24T09:15:33.817 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66773.log 2023-11-24T09:15:33.818 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56196.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48015.log.gz 2023-11-24T09:15:33.820 INFO:teuthology.orchestra.run.smithi164.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56196.log.gz 2023-11-24T09:15:33.821 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53040.log 2023-11-24T09:15:33.822 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47845.log 2023-11-24T09:15:33.822 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66773.log: /var/log/ceph/cluster1-client.mirror.53040.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66773.log.gz 2023-11-24T09:15:33.823 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65929.log 2023-11-24T09:15:33.823 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47845.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53040.log.gz 2023-11-24T09:15:33.824 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47845.log.gz 2023-11-24T09:15:33.824 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71530.log 2023-11-24T09:15:33.825 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54083.log 2023-11-24T09:15:33.825 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65929.log: /var/log/ceph/cluster1-client.mirror.71530.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71530.log.gz 2023-11-24T09:15:33.825 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65929.log.gz 2023-11-24T09:15:33.826 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48556.log 2023-11-24T09:15:33.826 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38957.log 2023-11-24T09:15:33.826 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54083.log: /var/log/ceph/cluster2-client.mirror.48556.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54083.log.gz 2023-11-24T09:15:33.827 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52183.log 2023-11-24T09:15:33.827 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.38957.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38957.log.gz 2023-11-24T09:15:33.828 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47956.log 2023-11-24T09:15:33.828 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52183.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52183.log.gz 2023-11-24T09:15:33.830 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47442.log 2023-11-24T09:15:33.830 INFO:teuthology.orchestra.run.smithi164.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.49785.log.gz 2023-11-24T09:15:33.830 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47956.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47956.log.gz 2023-11-24T09:15:33.830 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55113.log 2023-11-24T09:15:33.830 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64003.log 2023-11-24T09:15:33.831 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47442.log: /var/log/ceph/cluster2-client.mirror.55113.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55113.log.gz 2023-11-24T09:15:33.834 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62908.log 2023-11-24T09:15:33.835 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64003.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47442.log.gz 2023-11-24T09:15:33.836 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43677.log 2023-11-24T09:15:33.838 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62908.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64003.log.gz 2023-11-24T09:15:33.838 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62908.log.gz 2023-11-24T09:15:33.838 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50743.log 2023-11-24T09:15:33.838 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62307.log 2023-11-24T09:15:33.838 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50743.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50743.log.gz 2023-11-24T09:15:33.839 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63177.log 2023-11-24T09:15:33.839 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62307.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62307.log.gz 2023-11-24T09:15:33.840 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54037.log 2023-11-24T09:15:33.841 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63177.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63177.log.gz 2023-11-24T09:15:33.842 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56398.log 2023-11-24T09:15:33.842 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54037.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54037.log.gz 2023-11-24T09:15:33.843 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63913.log 2023-11-24T09:15:33.843 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56398.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43677.log.gz 2023-11-24T09:15:33.844 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.39821.log 2023-11-24T09:15:33.844 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63913.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56398.log.gz 2023-11-24T09:15:33.844 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43802.log 2023-11-24T09:15:33.845 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63913.log.gz 2023-11-24T09:15:33.845 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67795.log 2023-11-24T09:15:33.845 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.2.39821.log: /var/log/ceph/cluster1-client.admin.43802.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43802.log.gz 2023-11-24T09:15:33.846 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43501.log 2023-11-24T09:15:33.847 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67795.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67795.log.gz 2023-11-24T09:15:33.847 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65907.log 2023-11-24T09:15:33.848 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43501.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43501.log.gz 2023-11-24T09:15:33.848 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53571.log 2023-11-24T09:15:33.851 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65907.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65907.log.gz 2023-11-24T09:15:33.851 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46959.log 2023-11-24T09:15:33.852 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53571.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53571.log.gz 2023-11-24T09:15:33.852 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79556.log 2023-11-24T09:15:33.856 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46959.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52972.log.gz 2023-11-24T09:15:33.858 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.63611.log 2023-11-24T09:15:33.858 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79556.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79556.log.gz 2023-11-24T09:15:33.858 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46959.log.gz 2023-11-24T09:15:33.858 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40385.log 2023-11-24T09:15:33.859 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.2.63611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68650.log 2023-11-24T09:15:33.859 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.40385.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.40385.log.gz 2023-11-24T09:15:33.859 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75030.log 2023-11-24T09:15:33.860 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68650.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68650.log.gz 2023-11-24T09:15:33.860 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54851.log 2023-11-24T09:15:33.861 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75030.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75030.log.gz 2023-11-24T09:15:33.862 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55252.log 2023-11-24T09:15:33.864 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54851.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54851.log.gz 2023-11-24T09:15:33.865 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44820.log 2023-11-24T09:15:33.866 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55252.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55252.log.gz 2023-11-24T09:15:33.866 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45570.log 2023-11-24T09:15:33.871 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44820.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44820.log.gz 2023-11-24T09:15:33.871 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70585.log 2023-11-24T09:15:33.876 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.45570.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45570.log.gz 2023-11-24T09:15:33.877 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64858.log 2023-11-24T09:15:33.878 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70585.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70585.log.gz 2023-11-24T09:15:33.878 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42670.log 2023-11-24T09:15:33.879 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64858.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64858.log.gz 2023-11-24T09:15:33.879 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48514.log 2023-11-24T09:15:33.885 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42670.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42670.log.gz 2023-11-24T09:15:33.886 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79790.log 2023-11-24T09:15:33.887 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48514.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48514.log.gz 2023-11-24T09:15:33.887 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52163.log 2023-11-24T09:15:33.888 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79790.log.gz 2023-11-24T09:15:33.888 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43968.log 2023-11-24T09:15:33.892 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52163.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.63611.log.gz 2023-11-24T09:15:33.892 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49957.log 2023-11-24T09:15:33.893 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43968.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43968.log.gz 2023-11-24T09:15:33.894 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61638.log 2023-11-24T09:15:33.894 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49957.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49957.log.gz 2023-11-24T09:15:33.895 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.63002.log 2023-11-24T09:15:33.896 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61638.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61638.log.gz 2023-11-24T09:15:33.896 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55004.log 2023-11-24T09:15:33.910 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.63002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66368.log 2023-11-24T09:15:33.911 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55004.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55004.log.gz 2023-11-24T09:15:33.912 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61143.log 2023-11-24T09:15:33.912 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66368.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66368.log.gz 2023-11-24T09:15:33.926 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59991.log 2023-11-24T09:15:33.928 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61143.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61143.log.gz 2023-11-24T09:15:33.929 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.76080.log 2023-11-24T09:15:33.931 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59991.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59991.log.gz 2023-11-24T09:15:33.946 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.39949.log 2023-11-24T09:15:33.949 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.76080.log: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.39821.log.gz 2023-11-24T09:15:33.949 INFO:teuthology.orchestra.run.smithi164.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.76080.log.gz 2023-11-24T09:15:33.963 INFO:teuthology.orchestra.run.smithi164.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.63002.log.gz 2023-11-24T09:15:33.963 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52930.log 2023-11-24T09:15:33.963 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.3.39949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78984.log 2023-11-24T09:15:33.964 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52930.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52930.log.gz 2023-11-24T09:15:33.964 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49143.log 2023-11-24T09:15:33.965 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78984.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78984.log.gz 2023-11-24T09:15:33.965 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.39926.log 2023-11-24T09:15:33.967 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49143.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49143.log.gz 2023-11-24T09:15:33.967 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75701.log 2023-11-24T09:15:33.970 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.2.39926.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.39949.log.gz 2023-11-24T09:15:33.982 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39404.log 2023-11-24T09:15:33.983 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75701.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75701.log.gz 2023-11-24T09:15:33.984 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62088.log 2023-11-24T09:15:33.984 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39404.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39404.log.gz 2023-11-24T09:15:33.985 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45780.log 2023-11-24T09:15:33.987 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62088.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62088.log.gz 2023-11-24T09:15:33.988 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57920.log 2023-11-24T09:15:33.989 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.45780.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45780.log.gz 2023-11-24T09:15:33.989 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60641.log 2023-11-24T09:15:33.990 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57920.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57920.log.gz 2023-11-24T09:15:33.990 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74742.log 2023-11-24T09:15:33.991 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60641.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60641.log.gz 2023-11-24T09:15:33.991 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64397.log 2023-11-24T09:15:33.992 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74742.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74742.log.gz 2023-11-24T09:15:33.993 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62428.log 2023-11-24T09:15:33.994 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64397.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64397.log.gz 2023-11-24T09:15:33.994 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75559.log 2023-11-24T09:15:33.995 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62428.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62428.log.gz 2023-11-24T09:15:33.995 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53857.log 2023-11-24T09:15:33.996 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75559.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.75559.log.gz 2023-11-24T09:15:33.996 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75724.log 2023-11-24T09:15:33.997 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53857.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53857.log.gz 2023-11-24T09:15:33.998 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68175.log 2023-11-24T09:15:33.999 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75724.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75724.log.gz 2023-11-24T09:15:33.999 INFO:teuthology.orchestra.run.smithi164.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.39926.log.gz 2023-11-24T09:15:34.001 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52378.log 2023-11-24T09:15:34.003 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53352.log 2023-11-24T09:15:34.003 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68175.log: /var/log/ceph/cluster2-client.mirror.52378.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52378.log.gz 2023-11-24T09:15:34.003 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68175.log.gz 2023-11-24T09:15:34.003 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66593.log 2023-11-24T09:15:34.003 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62756.log 2023-11-24T09:15:34.003 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53352.log: /var/log/ceph/cluster1-client.mirror.66593.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53352.log.gz 2023-11-24T09:15:34.003 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66593.log.gz 2023-11-24T09:15:34.003 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75600.log 2023-11-24T09:15:34.004 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62021.log 2023-11-24T09:15:34.004 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62756.log: /var/log/ceph/cluster2-client.mirror.75600.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62756.log.gz 2023-11-24T09:15:34.004 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47645.log 2023-11-24T09:15:34.004 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62021.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62021.log.gz 2023-11-24T09:15:34.005 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51818.log 2023-11-24T09:15:34.005 INFO:teuthology.orchestra.run.smithi164.stderr: 91.2%/var/log/ceph/cluster2-client.mirror.47645.log: -- replaced with /var/log/ceph/cluster2-client.mirror.75600.log.gz 2023-11-24T09:15:34.007 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47645.log.gz 2023-11-24T09:15:34.007 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71035.log 2023-11-24T09:15:34.007 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75638.log 2023-11-24T09:15:34.007 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51818.log: /var/log/ceph/cluster1-client.mirror.71035.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51818.log.gz 2023-11-24T09:15:34.007 INFO:teuthology.orchestra.run.smithi164.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71035.log.gz 2023-11-24T09:15:34.007 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56883.log 2023-11-24T09:15:34.007 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62497.log 2023-11-24T09:15:34.008 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75638.log: /var/log/ceph/cluster2-client.mirror.56883.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56883.log.gz 2023-11-24T09:15:34.008 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75638.log.gz 2023-11-24T09:15:34.009 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49583.log 2023-11-24T09:15:34.009 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66550.log 2023-11-24T09:15:34.010 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62497.log: /var/log/ceph/cluster2-client.mirror.49583.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62497.log.gz 2023-11-24T09:15:34.010 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49583.log.gz 2023-11-24T09:15:34.010 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77632.log 2023-11-24T09:15:34.011 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76619.log 2023-11-24T09:15:34.011 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66550.log: /var/log/ceph/cluster2-client.admin.77632.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77632.log.gz 2023-11-24T09:15:34.012 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66550.log.gz 2023-11-24T09:15:34.012 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75658.log 2023-11-24T09:15:34.012 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57405.log 2023-11-24T09:15:34.013 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76619.log: /var/log/ceph/cluster2-client.mirror.75658.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76619.log.gz 2023-11-24T09:15:34.013 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75658.log.gz 2023-11-24T09:15:34.013 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74517.log 2023-11-24T09:15:34.013 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57405.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57405.log.gz 2023-11-24T09:15:34.014 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46220.log 2023-11-24T09:15:34.014 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58700.log 2023-11-24T09:15:34.014 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74517.log: /var/log/ceph/cluster1-client.mirror.46220.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74517.log.gz 2023-11-24T09:15:34.015 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77658.log 2023-11-24T09:15:34.015 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58700.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75195.log.gz 2023-11-24T09:15:34.017 INFO:teuthology.orchestra.run.smithi164.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58700.log.gz 2023-11-24T09:15:34.017 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55788.log 2023-11-24T09:15:34.018 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65704.log 2023-11-24T09:15:34.018 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77658.log: /var/log/ceph/cluster1-client.mirror.55788.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77658.log.gz 2023-11-24T09:15:34.018 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43132.log 2023-11-24T09:15:34.019 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65704.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46220.log.gz 2023-11-24T09:15:34.019 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65704.log.gz 2023-11-24T09:15:34.019 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55788.log.gz 2023-11-24T09:15:34.019 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39534.log 2023-11-24T09:15:34.020 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63155.log 2023-11-24T09:15:34.021 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.49299.log 2023-11-24T09:15:34.021 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39534.log.gz 2023-11-24T09:15:34.022 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63155.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63155.log.gz 2023-11-24T09:15:34.022 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45612.log 2023-11-24T09:15:34.023 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51108.log 2023-11-24T09:15:34.024 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.1.49299.log: /var/log/ceph/cluster2-client.mirror.45612.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45612.log.gz 2023-11-24T09:15:34.039 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39895.log 2023-11-24T09:15:34.040 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51108.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51108.log.gz 2023-11-24T09:15:34.040 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47073.log 2023-11-24T09:15:34.041 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.39895.log: 73.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39895.log.gz 2023-11-24T09:15:34.041 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43856.log 2023-11-24T09:15:34.048 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47073.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47073.log.gz 2023-11-24T09:15:34.048 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50182.log 2023-11-24T09:15:34.050 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43856.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43856.log.gz 2023-11-24T09:15:34.050 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46638.log 2023-11-24T09:15:34.051 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.49299.log.gz 2023-11-24T09:15:34.067 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67042.log 2023-11-24T09:15:34.073 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46638.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46638.log.gz 2023-11-24T09:15:34.073 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.39842.log 2023-11-24T09:15:34.076 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67042.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67042.log.gz 2023-11-24T09:15:34.076 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57382.log 2023-11-24T09:15:34.084 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.39842.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75538.log 2023-11-24T09:15:34.085 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57382.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57382.log.gz 2023-11-24T09:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.39842.log.gz 2023-11-24T09:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46110.log 2023-11-24T09:15:34.087 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.75538.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75538.log.gz 2023-11-24T09:15:34.087 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71823.log 2023-11-24T09:15:34.088 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46110.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46110.log.gz 2023-11-24T09:15:34.088 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50785.log 2023-11-24T09:15:34.089 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71823.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71823.log.gz 2023-11-24T09:15:34.089 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57583.log 2023-11-24T09:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47705.log 2023-11-24T09:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57583.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57583.log.gz 2023-11-24T09:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76427.log.gz 2023-11-24T09:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65354.log 2023-11-24T09:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47705.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47705.log.gz 2023-11-24T09:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49425.log 2023-11-24T09:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65354.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65354.log.gz 2023-11-24T09:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58444.log 2023-11-24T09:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49425.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49425.log.gz 2023-11-24T09:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68423.log 2023-11-24T09:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58444.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58444.log.gz 2023-11-24T09:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74538.log 2023-11-24T09:15:34.117 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68423.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68423.log.gz 2023-11-24T09:15:34.117 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56619.log 2023-11-24T09:15:34.119 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74538.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74538.log.gz 2023-11-24T09:15:34.119 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41647.log 2023-11-24T09:15:34.120 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56619.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56619.log.gz 2023-11-24T09:15:34.121 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61459.log 2023-11-24T09:15:34.128 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.41647.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.41647.log.gz 2023-11-24T09:15:34.128 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75977.log 2023-11-24T09:15:34.129 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61459.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61459.log.gz 2023-11-24T09:15:34.129 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69733.log 2023-11-24T09:15:34.132 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75977.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52163.log.gz 2023-11-24T09:15:34.132 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74788.log 2023-11-24T09:15:34.134 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69733.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69733.log.gz 2023-11-24T09:15:34.134 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52833.log 2023-11-24T09:15:34.135 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74788.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.74788.log.gz 2023-11-24T09:15:34.135 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54262.log 2023-11-24T09:15:34.136 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52833.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52833.log.gz 2023-11-24T09:15:34.136 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68381.log 2023-11-24T09:15:34.137 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54262.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54262.log.gz 2023-11-24T09:15:34.138 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39818.log 2023-11-24T09:15:34.140 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68381.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68381.log.gz 2023-11-24T09:15:34.142 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48419.log 2023-11-24T09:15:34.143 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.39818.log: 83.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39818.log.gz 2023-11-24T09:15:34.143 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65228.log 2023-11-24T09:15:34.145 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48419.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48419.log.gz 2023-11-24T09:15:34.145 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59950.log 2023-11-24T09:15:34.147 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65228.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65228.log.gz 2023-11-24T09:15:34.148 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56952.log 2023-11-24T09:15:34.149 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59950.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59950.log.gz 2023-11-24T09:15:34.149 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57874.log 2023-11-24T09:15:34.150 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56952.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56952.log.gz 2023-11-24T09:15:34.150 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59971.log 2023-11-24T09:15:34.151 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57874.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57874.log.gz 2023-11-24T09:15:34.151 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64045.log 2023-11-24T09:15:34.154 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59971.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59971.log.gz 2023-11-24T09:15:34.154 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69919.log 2023-11-24T09:15:34.155 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64045.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64045.log.gz 2023-11-24T09:15:34.155 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54695.log 2023-11-24T09:15:34.156 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69919.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69919.log.gz 2023-11-24T09:15:34.157 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63002.log 2023-11-24T09:15:34.158 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54695.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54695.log.gz 2023-11-24T09:15:34.158 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52242.log 2023-11-24T09:15:34.171 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49526.log 2023-11-24T09:15:34.186 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49163.log 2023-11-24T09:15:34.203 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49526.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63002.log.gz 2023-11-24T09:15:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52759.log 2023-11-24T09:15:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49163.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49163.log.gz 2023-11-24T09:15:34.234 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42101.log 2023-11-24T09:15:34.242 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51667.log 2023-11-24T09:15:34.243 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42101.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42101.log.gz 2023-11-24T09:15:34.250 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48815.log.gz 2023-11-24T09:15:34.254 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47725.log 2023-11-24T09:15:34.271 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57695.log 2023-11-24T09:15:34.271 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47725.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47725.log.gz 2023-11-24T09:15:34.272 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66685.log 2023-11-24T09:15:34.272 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57695.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57695.log.gz 2023-11-24T09:15:34.283 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61010.log 2023-11-24T09:15:34.284 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66685.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66685.log.gz 2023-11-24T09:15:34.299 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40543.log 2023-11-24T09:15:34.299 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61010.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61010.log.gz 2023-11-24T09:15:34.314 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54583.log 2023-11-24T09:15:34.315 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40543.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40543.log.gz 2023-11-24T09:15:34.315 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77710.log 2023-11-24T09:15:34.317 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54583.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54583.log.gz 2023-11-24T09:15:34.331 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64712.log 2023-11-24T09:15:34.332 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77710.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77710.log.gz 2023-11-24T09:15:34.332 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74350.log 2023-11-24T09:15:34.333 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64712.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64712.log.gz 2023-11-24T09:15:34.346 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55339.log 2023-11-24T09:15:34.361 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74350.log: 93.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69175.log 2023-11-24T09:15:34.364 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55339.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43132.log.gz 2023-11-24T09:15:34.364 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55339.log.gz 2023-11-24T09:15:34.364 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60754.log 2023-11-24T09:15:34.365 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69175.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69175.log.gz 2023-11-24T09:15:34.365 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63936.log 2023-11-24T09:15:34.367 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60754.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60754.log.gz 2023-11-24T09:15:34.367 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66033.log 2023-11-24T09:15:34.369 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63936.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63936.log.gz 2023-11-24T09:15:34.369 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55442.log 2023-11-24T09:15:34.369 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66033.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66033.log.gz 2023-11-24T09:15:34.369 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67857.log 2023-11-24T09:15:34.371 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55442.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55442.log.gz 2023-11-24T09:15:34.371 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51419.log 2023-11-24T09:15:34.371 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67857.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67857.log.gz 2023-11-24T09:15:34.372 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59411.log 2023-11-24T09:15:34.387 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50726.log 2023-11-24T09:15:34.388 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59411.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59411.log.gz 2023-11-24T09:15:34.416 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43297.log 2023-11-24T09:15:34.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50726.log: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50726.log.gz 2023-11-24T09:15:34.416 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47116.log 2023-11-24T09:15:34.417 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43297.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43297.log.gz 2023-11-24T09:15:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64484.log 2023-11-24T09:15:34.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47116.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48556.log.gz 2023-11-24T09:15:34.440 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47116.log.gz 2023-11-24T09:15:34.450 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79582.log 2023-11-24T09:15:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64484.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64484.log.gz 2023-11-24T09:15:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67208.log 2023-11-24T09:15:34.468 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79582.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79582.log.gz 2023-11-24T09:15:34.468 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75977.log.gz 2023-11-24T09:15:34.474 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50182.log.gz 2023-11-24T09:15:34.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74830.log 2023-11-24T09:15:34.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67208.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67208.log.gz 2023-11-24T09:15:34.476 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64556.log 2023-11-24T09:15:34.477 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74830.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74830.log.gz 2023-11-24T09:15:34.478 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63723.log 2023-11-24T09:15:34.479 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64556.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64556.log.gz 2023-11-24T09:15:34.479 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48905.log 2023-11-24T09:15:34.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63723.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63723.log.gz 2023-11-24T09:15:34.480 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58880.log 2023-11-24T09:15:34.481 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48905.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48905.log.gz 2023-11-24T09:15:34.482 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51704.log 2023-11-24T09:15:34.483 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58880.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58880.log.gz 2023-11-24T09:15:34.483 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68113.log 2023-11-24T09:15:34.485 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51704.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51704.log.gz 2023-11-24T09:15:34.485 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72003.log 2023-11-24T09:15:34.486 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68113.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68113.log.gz 2023-11-24T09:15:34.486 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50229.log 2023-11-24T09:15:34.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72003.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.72003.log.gz 2023-11-24T09:15:34.488 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62800.log 2023-11-24T09:15:34.489 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50229.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50229.log.gz 2023-11-24T09:15:34.490 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39326.log 2023-11-24T09:15:34.491 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62800.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62800.log.gz 2023-11-24T09:15:34.491 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44798.log 2023-11-24T09:15:34.492 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39326.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39326.log.gz 2023-11-24T09:15:34.492 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59321.log 2023-11-24T09:15:34.494 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44798.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44798.log.gz 2023-11-24T09:15:34.494 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74185.log 2023-11-24T09:15:34.495 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59321.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59321.log.gz 2023-11-24T09:15:34.496 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43088.log 2023-11-24T09:15:34.497 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74185.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74185.log.gz 2023-11-24T09:15:34.497 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42491.log 2023-11-24T09:15:34.500 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43088.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49526.log.gz 2023-11-24T09:15:34.503 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72183.log 2023-11-24T09:15:34.503 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50351.log 2023-11-24T09:15:34.503 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63847.log 2023-11-24T09:15:34.504 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42491.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42491.log.gz 2023-11-24T09:15:34.505 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43088.log.gz 2023-11-24T09:15:34.510 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50351.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50785.log.gz 2023-11-24T09:15:34.510 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72183.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72183.log.gz 2023-11-24T09:15:34.510 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57718.log 2023-11-24T09:15:34.511 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63847.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63847.log.gz 2023-11-24T09:15:34.512 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63135.log 2023-11-24T09:15:34.513 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57718.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57718.log.gz 2023-11-24T09:15:34.513 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57428.log 2023-11-24T09:15:34.515 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63135.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63135.log.gz 2023-11-24T09:15:34.515 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61886.log 2023-11-24T09:15:34.516 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57428.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57428.log.gz 2023-11-24T09:15:34.516 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60682.log 2023-11-24T09:15:34.517 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61886.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61886.log.gz 2023-11-24T09:15:34.518 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62108.log 2023-11-24T09:15:34.520 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60682.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60682.log.gz 2023-11-24T09:15:34.520 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58179.log 2023-11-24T09:15:34.522 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.62108.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62108.log.gz 2023-11-24T09:15:34.523 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64289.log 2023-11-24T09:15:34.523 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58179.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58179.log.gz 2023-11-24T09:15:34.524 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49064.log 2023-11-24T09:15:34.525 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64289.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64289.log.gz 2023-11-24T09:15:34.525 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66500.log 2023-11-24T09:15:34.526 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49064.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49064.log.gz 2023-11-24T09:15:34.527 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61269.log 2023-11-24T09:15:34.528 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66500.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66500.log.gz 2023-11-24T09:15:34.528 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72377.log 2023-11-24T09:15:34.529 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61269.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61269.log.gz 2023-11-24T09:15:34.529 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42869.log 2023-11-24T09:15:34.530 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.72377.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72377.log.gz 2023-11-24T09:15:34.530 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53374.log 2023-11-24T09:15:34.532 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42869.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42869.log.gz 2023-11-24T09:15:34.532 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54146.log 2023-11-24T09:15:34.540 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53374.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52242.log.gz 2023-11-24T09:15:34.541 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57739.log 2023-11-24T09:15:34.542 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54146.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54146.log.gz 2023-11-24T09:15:34.542 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67249.log 2023-11-24T09:15:34.544 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57739.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57739.log.gz 2023-11-24T09:15:34.544 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50373.log 2023-11-24T09:15:34.546 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67249.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67249.log.gz 2023-11-24T09:15:34.546 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72025.log 2023-11-24T09:15:34.549 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50373.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50373.log.gz 2023-11-24T09:15:34.550 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48968.log 2023-11-24T09:15:34.551 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72025.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72025.log.gz 2023-11-24T09:15:34.551 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47505.log 2023-11-24T09:15:34.551 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48968.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48968.log.gz 2023-11-24T09:15:34.552 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54915.log 2023-11-24T09:15:34.553 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47505.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47505.log.gz 2023-11-24T09:15:34.553 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49933.log 2023-11-24T09:15:34.554 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.54915.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54915.log.gz 2023-11-24T09:15:34.555 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43045.log 2023-11-24T09:15:34.566 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49933.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50103.log 2023-11-24T09:15:34.583 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43045.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71620.log 2023-11-24T09:15:34.583 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43045.log.gz 2023-11-24T09:15:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39949.log 2023-11-24T09:15:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71620.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71620.log.gz 2023-11-24T09:15:34.607 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72295.log 2023-11-24T09:15:34.607 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.39949.log: 81.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39949.log.gz 2023-11-24T09:15:34.615 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44564.log 2023-11-24T09:15:34.615 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72295.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72295.log.gz 2023-11-24T09:15:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44431.log 2023-11-24T09:15:34.630 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44564.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59750.log 2023-11-24T09:15:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44431.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44431.log.gz 2023-11-24T09:15:34.638 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47422.log 2023-11-24T09:15:34.640 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59750.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59750.log.gz 2023-11-24T09:15:34.642 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44564.log.gz 2023-11-24T09:15:34.651 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71463.log 2023-11-24T09:15:34.659 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47422.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55298.log 2023-11-24T09:15:34.659 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71463.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71463.log.gz 2023-11-24T09:15:34.667 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51608.log 2023-11-24T09:15:34.667 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55298.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55298.log.gz 2023-11-24T09:15:34.670 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47422.log.gz 2023-11-24T09:15:34.674 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74350.log.gz 2023-11-24T09:15:34.674 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52084.log 2023-11-24T09:15:34.675 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51608.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51608.log.gz 2023-11-24T09:15:34.675 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63091.log 2023-11-24T09:15:34.691 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57238.log 2023-11-24T09:15:34.692 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63091.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63091.log.gz 2023-11-24T09:15:34.706 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60320.log 2023-11-24T09:15:34.707 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57238.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57238.log.gz 2023-11-24T09:15:34.712 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52759.log.gz 2023-11-24T09:15:34.727 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65164.log 2023-11-24T09:15:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60320.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60320.log.gz 2023-11-24T09:15:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47245.log 2023-11-24T09:15:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65164.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65164.log.gz 2023-11-24T09:15:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64443.log 2023-11-24T09:15:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47245.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44014.log 2023-11-24T09:15:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64443.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64443.log.gz 2023-11-24T09:15:34.788 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51667.log.gz 2023-11-24T09:15:34.788 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46981.log 2023-11-24T09:15:34.789 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44014.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44014.log.gz 2023-11-24T09:15:34.790 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70090.log 2023-11-24T09:15:34.791 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46981.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47245.log.gz 2023-11-24T09:15:34.791 INFO:teuthology.orchestra.run.smithi164.stderr: 79.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46981.log.gz 2023-11-24T09:15:34.791 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58524.log 2023-11-24T09:15:34.791 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70090.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70090.log.gz 2023-11-24T09:15:34.792 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70315.log 2023-11-24T09:15:34.793 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58524.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58524.log.gz 2023-11-24T09:15:34.794 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46087.log 2023-11-24T09:15:34.794 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70315.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70315.log.gz 2023-11-24T09:15:34.794 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72205.log 2023-11-24T09:15:34.795 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46087.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46087.log.gz 2023-11-24T09:15:34.795 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71868.log 2023-11-24T09:15:34.796 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72205.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72205.log.gz 2023-11-24T09:15:34.796 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43321.log 2023-11-24T09:15:34.797 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71868.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71868.log.gz 2023-11-24T09:15:34.797 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59100.log 2023-11-24T09:15:34.799 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43321.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43321.log.gz 2023-11-24T09:15:34.799 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64732.log 2023-11-24T09:15:34.800 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59100.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59100.log.gz 2023-11-24T09:15:34.800 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73523.log 2023-11-24T09:15:34.802 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64732.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64732.log.gz 2023-11-24T09:15:34.803 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55701.log 2023-11-24T09:15:34.804 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.73523.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73523.log.gz 2023-11-24T09:15:34.804 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67898.log 2023-11-24T09:15:34.805 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.55701.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55701.log.gz 2023-11-24T09:15:34.805 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60257.log 2023-11-24T09:15:34.806 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67898.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67898.log.gz 2023-11-24T09:15:34.807 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60506.log 2023-11-24T09:15:34.808 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60257.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60257.log.gz 2023-11-24T09:15:34.808 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72394.log 2023-11-24T09:15:34.809 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60506.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60506.log.gz 2023-11-24T09:15:34.810 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47159.log 2023-11-24T09:15:34.811 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.72394.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72394.log.gz 2023-11-24T09:15:34.811 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64506.log 2023-11-24T09:15:34.818 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47159.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47159.log.gz 2023-11-24T09:15:34.818 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67629.log 2023-11-24T09:15:34.819 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64506.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64506.log.gz 2023-11-24T09:15:34.819 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67547.log 2023-11-24T09:15:34.822 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67629.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.67629.log.gz 2023-11-24T09:15:34.822 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71350.log 2023-11-24T09:15:34.822 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62328.log 2023-11-24T09:15:34.824 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51419.log.gz/var/log/ceph/cluster1-client.mirror.71350.log: 2023-11-24T09:15:34.824 INFO:teuthology.orchestra.run.smithi164.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71350.log.gz 2023-11-24T09:15:34.824 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56148.log 2023-11-24T09:15:34.824 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.62328.log: /var/log/ceph/cluster2-client.mirror.67547.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67547.log.gz 2023-11-24T09:15:34.825 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60774.log 2023-11-24T09:15:34.826 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56148.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56148.log.gz 2023-11-24T09:15:34.826 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62328.log.gz 2023-11-24T09:15:34.826 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58857.log 2023-11-24T09:15:34.827 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66170.log 2023-11-24T09:15:34.827 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.60774.log: /var/log/ceph/cluster1-client.mirror.58857.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60774.log.gz 2023-11-24T09:15:34.827 INFO:teuthology.orchestra.run.smithi164.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58857.log.gz 2023-11-24T09:15:34.827 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74208.log 2023-11-24T09:15:34.828 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67989.log 2023-11-24T09:15:34.828 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66170.log: /var/log/ceph/cluster1-client.mirror.74208.log: 86.9% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66170.log.gz 2023-11-24T09:15:34.828 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.74208.log.gz 2023-11-24T09:15:34.829 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76536.log 2023-11-24T09:15:34.829 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71643.log 2023-11-24T09:15:34.830 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67989.log: /var/log/ceph/cluster1-client.mirror.76536.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76536.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67989.log.gz 2023-11-24T09:15:34.830 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:34.830 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56754.log 2023-11-24T09:15:34.832 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40402.log 2023-11-24T09:15:34.832 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71643.log: /var/log/ceph/cluster1-client.mirror.56754.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71643.log.gz 2023-11-24T09:15:34.832 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56754.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54810.log 2023-11-24T09:15:34.832 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:15:34.832 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.40402.log: 67.2% -- replaced with /var/log/ceph/cluster2-client.admin.40402.log.gz 2023-11-24T09:15:34.832 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58354.log 2023-11-24T09:15:34.832 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61315.log 2023-11-24T09:15:34.833 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54810.log: /var/log/ceph/cluster1-client.mirror.58354.log: 87.6% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58354.log.gz 2023-11-24T09:15:34.833 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54810.log.gz 2023-11-24T09:15:34.834 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51625.log 2023-11-24T09:15:34.834 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39334.log 2023-11-24T09:15:34.834 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61315.log: /var/log/ceph/cluster2-client.mirror.51625.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61315.log.gz 2023-11-24T09:15:34.836 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64622.log 2023-11-24T09:15:34.836 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51625.log.gz 2023-11-24T09:15:34.836 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.39334.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39334.log.gz 2023-11-24T09:15:34.836 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70990.log 2023-11-24T09:15:34.836 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71125.log 2023-11-24T09:15:34.837 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64622.log: /var/log/ceph/cluster1-client.mirror.70990.log: 86.7% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64622.log.gz 2023-11-24T09:15:34.838 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70990.log.gz 2023-11-24T09:15:34.838 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51809.log 2023-11-24T09:15:34.838 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76557.log 2023-11-24T09:15:34.838 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71125.log: /var/log/ceph/cluster1-client.mirror.51809.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71125.log.gz 2023-11-24T09:15:34.838 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78152.log 2023-11-24T09:15:34.839 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76557.log: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51809.log.gz 2023-11-24T09:15:34.839 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76557.log.gz 2023-11-24T09:15:34.839 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40299.log 2023-11-24T09:15:34.839 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76315.log 2023-11-24T09:15:34.841 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78152.log: /var/log/ceph/cluster1-client.mirror.40299.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78152.log.gz 2023-11-24T09:15:34.841 INFO:teuthology.orchestra.run.smithi164.stderr: 56.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40299.log.gz 2023-11-24T09:15:34.841 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55206.log 2023-11-24T09:15:34.841 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69960.log 2023-11-24T09:15:34.841 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76315.log: /var/log/ceph/cluster1-client.mirror.55206.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.76315.log.gz 2023-11-24T09:15:34.842 INFO:teuthology.orchestra.run.smithi164.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55206.log.gz 2023-11-24T09:15:34.842 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70293.log 2023-11-24T09:15:34.842 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52581.log 2023-11-24T09:15:34.842 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69960.log: /var/log/ceph/cluster1-client.mirror.70293.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70293.log.gz 2023-11-24T09:15:34.843 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69960.log.gz 2023-11-24T09:15:34.844 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41712.log 2023-11-24T09:15:34.845 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71080.log 2023-11-24T09:15:34.866 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52581.log: /var/log/ceph/cluster2-client.mirror.41712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53877.log 2023-11-24T09:15:34.867 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71080.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71080.log.gz 2023-11-24T09:15:34.886 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53374.log.gz 2023-11-24T09:15:34.886 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61932.log 2023-11-24T09:15:34.887 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53877.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53877.log.gz 2023-11-24T09:15:34.887 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56795.log 2023-11-24T09:15:34.888 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61932.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61932.log.gz 2023-11-24T09:15:34.889 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79608.log 2023-11-24T09:15:34.890 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56795.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56795.log.gz 2023-11-24T09:15:34.891 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70855.log 2023-11-24T09:15:34.891 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79608.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79608.log.gz 2023-11-24T09:15:34.892 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.51172.log 2023-11-24T09:15:34.892 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70855.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70855.log.gz 2023-11-24T09:15:34.892 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46572.log 2023-11-24T09:15:34.903 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.51172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69051.log 2023-11-24T09:15:34.908 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46572.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46572.log.gz 2023-11-24T09:15:34.911 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50351.log.gz 2023-11-24T09:15:34.911 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46378.log 2023-11-24T09:15:34.912 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69051.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69051.log.gz 2023-11-24T09:15:34.913 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78906.log 2023-11-24T09:15:34.914 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46378.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46378.log.gz 2023-11-24T09:15:34.914 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66918.log 2023-11-24T09:15:34.915 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78906.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78906.log.gz 2023-11-24T09:15:34.915 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48195.log 2023-11-24T09:15:34.918 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66918.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66918.log.gz 2023-11-24T09:15:34.918 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78100.log 2023-11-24T09:15:34.925 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48195.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48195.log.gz 2023-11-24T09:15:34.925 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40504.log 2023-11-24T09:15:34.925 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78100.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78100.log.gz 2023-11-24T09:15:34.925 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71665.log 2023-11-24T09:15:34.925 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40504.log: 55.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40504.log.gz 2023-11-24T09:15:34.925 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74140.log 2023-11-24T09:15:34.925 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71665.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71665.log.gz 2023-11-24T09:15:34.925 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49721.log 2023-11-24T09:15:34.926 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74140.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74140.log.gz 2023-11-24T09:15:34.926 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59298.log 2023-11-24T09:15:34.926 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49721.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49721.log.gz 2023-11-24T09:15:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57192.log 2023-11-24T09:15:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59298.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59298.log.gz 2023-11-24T09:15:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54539.log 2023-11-24T09:15:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.57192.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57192.log.gz 2023-11-24T09:15:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59167.log 2023-11-24T09:15:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54539.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54539.log.gz 2023-11-24T09:15:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49563.log 2023-11-24T09:15:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59167.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59167.log.gz 2023-11-24T09:15:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57851.log 2023-11-24T09:15:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49563.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49563.log.gz 2023-11-24T09:15:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55163.log 2023-11-24T09:15:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.57851.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57851.log.gz 2023-11-24T09:15:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53664.log 2023-11-24T09:15:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.55163.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55163.log.gz 2023-11-24T09:15:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69816.log 2023-11-24T09:15:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53664.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53664.log.gz 2023-11-24T09:15:34.936 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47585.log 2023-11-24T09:15:34.936 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69816.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69816.log.gz 2023-11-24T09:15:34.937 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53710.log 2023-11-24T09:15:34.937 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47585.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47585.log.gz 2023-11-24T09:15:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76782.log 2023-11-24T09:15:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53710.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53710.log.gz 2023-11-24T09:15:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46443.log 2023-11-24T09:15:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76782.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.76782.log.gz 2023-11-24T09:15:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43991.log 2023-11-24T09:15:34.946 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46443.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46443.log.gz 2023-11-24T09:15:34.946 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65754.log 2023-11-24T09:15:34.947 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.43991.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43991.log.gz 2023-11-24T09:15:34.947 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65526.log 2023-11-24T09:15:34.948 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65754.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65754.log.gz 2023-11-24T09:15:34.949 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78646.log 2023-11-24T09:15:34.949 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.65526.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65526.log.gz 2023-11-24T09:15:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50958.log 2023-11-24T09:15:34.950 INFO:teuthology.orchestra.run.smithi164.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.51172.log.gz 2023-11-24T09:15:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78646.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78646.log.gz 2023-11-24T09:15:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56817.log 2023-11-24T09:15:34.951 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50958.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50921.log 2023-11-24T09:15:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.56817.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50958.log.gz 2023-11-24T09:15:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69878.log 2023-11-24T09:15:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50921.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50921.log.gz 2023-11-24T09:15:34.953 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56817.log.gz 2023-11-24T09:15:34.953 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70069.log 2023-11-24T09:15:34.953 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52062.log 2023-11-24T09:15:34.953 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70069.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69878.log.gz 2023-11-24T09:15:34.953 INFO:teuthology.orchestra.run.smithi164.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70069.log.gz 2023-11-24T09:15:34.953 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71733.log 2023-11-24T09:15:34.955 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67815.log 2023-11-24T09:15:34.955 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71733.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52062.log.gz 2023-11-24T09:15:34.955 INFO:teuthology.orchestra.run.smithi164.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71733.log.gz 2023-11-24T09:15:34.955 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66980.log 2023-11-24T09:15:34.957 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67815.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67815.log.gz 2023-11-24T09:15:34.957 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74229.log 2023-11-24T09:15:34.957 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47825.log 2023-11-24T09:15:34.958 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74229.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74229.log.gz 2023-11-24T09:15:34.960 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66980.log.gz 2023-11-24T09:15:34.960 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65187.log 2023-11-24T09:15:34.960 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79478.log 2023-11-24T09:15:34.961 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65187.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47825.log.gz 2023-11-24T09:15:34.961 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65187.log.gz 2023-11-24T09:15:34.961 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64226.log 2023-11-24T09:15:34.963 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42734.log 2023-11-24T09:15:34.963 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79478.log.gz 2023-11-24T09:15:34.963 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49005.log 2023-11-24T09:15:34.963 INFO:teuthology.orchestra.run.smithi164.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.42734.log: -- replaced with /var/log/ceph/cluster1-client.mirror.64226.log.gz 2023-11-24T09:15:34.982 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50605.log 2023-11-24T09:15:34.983 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49005.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49005.log.gz 2023-11-24T09:15:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75173.log 2023-11-24T09:15:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50605.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50605.log.gz 2023-11-24T09:15:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58464.log 2023-11-24T09:15:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75173.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75173.log.gz 2023-11-24T09:15:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67125.log 2023-11-24T09:15:34.988 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58464.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58464.log.gz 2023-11-24T09:15:34.988 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76403.log 2023-11-24T09:15:34.989 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67125.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67125.log.gz 2023-11-24T09:15:34.990 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51687.log 2023-11-24T09:15:34.991 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76403.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76403.log.gz 2023-11-24T09:15:34.991 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46200.log 2023-11-24T09:15:34.992 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51687.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51687.log.gz 2023-11-24T09:15:34.992 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51377.log 2023-11-24T09:15:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46200.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46200.log.gz 2023-11-24T09:15:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42647.log 2023-11-24T09:15:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51377.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51377.log.gz 2023-11-24T09:15:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59074.log 2023-11-24T09:15:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42647.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42647.log.gz 2023-11-24T09:15:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76014.log 2023-11-24T09:15:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59074.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59074.log.gz 2023-11-24T09:15:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58007.log 2023-11-24T09:15:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76014.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76014.log.gz 2023-11-24T09:15:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59569.log 2023-11-24T09:15:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.58007.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58007.log.gz 2023-11-24T09:15:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76828.log 2023-11-24T09:15:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59569.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59569.log.gz 2023-11-24T09:15:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67423.log 2023-11-24T09:15:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76828.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76828.log.gz 2023-11-24T09:15:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49662.log 2023-11-24T09:15:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67423.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67423.log.gz 2023-11-24T09:15:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70765.log 2023-11-24T09:15:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49662.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49662.log.gz 2023-11-24T09:15:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50160.log 2023-11-24T09:15:35.015 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70765.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70765.log.gz 2023-11-24T09:15:35.015 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61246.log 2023-11-24T09:15:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50160.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50160.log.gz 2023-11-24T09:15:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72115.log 2023-11-24T09:15:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61246.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61246.log.gz 2023-11-24T09:15:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-11-24T09:15:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72115.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.72115.log.gz 2023-11-24T09:15:35.019 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73926.log 2023-11-24T09:15:35.035 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40470.log 2023-11-24T09:15:35.035 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52084.log.gz 2023-11-24T09:15:35.035 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.73926.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73926.log.gz 2023-11-24T09:15:35.036 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46285.log 2023-11-24T09:15:35.036 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.40470.log: 58.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40470.log.gz 2023-11-24T09:15:35.036 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46486.log 2023-11-24T09:15:35.038 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46285.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46285.log.gz 2023-11-24T09:15:35.038 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44584.log 2023-11-24T09:15:35.044 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.46486.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.46486.log.gz 2023-11-24T09:15:35.044 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53756.log 2023-11-24T09:15:35.050 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44584.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44584.log.gz 2023-11-24T09:15:35.050 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64247.log 2023-11-24T09:15:35.051 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53756.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53756.log.gz 2023-11-24T09:15:35.052 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76037.log 2023-11-24T09:15:35.054 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.64247.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64247.log.gz 2023-11-24T09:15:35.054 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69134.log 2023-11-24T09:15:35.055 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76037.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76037.log.gz 2023-11-24T09:15:35.055 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43614.log 2023-11-24T09:15:35.056 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69134.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69134.log.gz 2023-11-24T09:15:35.056 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77187.log 2023-11-24T09:15:35.062 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.43614.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43614.log.gz 2023-11-24T09:15:35.062 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44143.log 2023-11-24T09:15:35.062 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49933.log.gz 2023-11-24T09:15:35.063 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.77187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41904.log 2023-11-24T09:15:35.064 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.44143.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44143.log.gz 2023-11-24T09:15:35.064 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66322.log 2023-11-24T09:15:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41904.log: 81.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41904.log.gz 2023-11-24T09:15:35.067 INFO:teuthology.orchestra.run.smithi164.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.77187.log.gz 2023-11-24T09:15:35.067 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78724.log 2023-11-24T09:15:35.067 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45271.log 2023-11-24T09:15:35.067 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78724.log: 87.7% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78724.log.gz 2023-11-24T09:15:35.067 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.66322.log.gz 2023-11-24T09:15:35.067 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48461.log 2023-11-24T09:15:35.068 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.45271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42935.log 2023-11-24T09:15:35.068 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.48461.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.45271.log.gz 2023-11-24T09:15:35.068 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77433.log 2023-11-24T09:15:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42935.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42935.log.gz 2023-11-24T09:15:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41298.log 2023-11-24T09:15:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77433.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77433.log.gz 2023-11-24T09:15:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63436.log 2023-11-24T09:15:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.41298.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.41298.log.gz 2023-11-24T09:15:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77580.log 2023-11-24T09:15:35.073 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63436.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63436.log.gz 2023-11-24T09:15:35.074 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76193.log 2023-11-24T09:15:35.074 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77580.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77580.log.gz 2023-11-24T09:15:35.074 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72048.log 2023-11-24T09:15:35.075 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76193.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76193.log.gz 2023-11-24T09:15:35.075 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50568.log 2023-11-24T09:15:35.076 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.72048.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72048.log.gz 2023-11-24T09:15:35.076 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63303.log 2023-11-24T09:15:35.077 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50568.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50568.log.gz 2023-11-24T09:15:35.077 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46810.log 2023-11-24T09:15:35.078 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63303.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63303.log.gz 2023-11-24T09:15:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75856.log 2023-11-24T09:15:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.46810.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46810.log.gz 2023-11-24T09:15:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42124.log 2023-11-24T09:15:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75856.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.75856.log.gz 2023-11-24T09:15:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49084.log 2023-11-24T09:15:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42124.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42124.log.gz 2023-11-24T09:15:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49299.log 2023-11-24T09:15:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49084.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49084.log.gz 2023-11-24T09:15:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78230.log 2023-11-24T09:15:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.49299.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49299.log.gz 2023-11-24T09:15:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74696.log 2023-11-24T09:15:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78230.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78230.log.gz 2023-11-24T09:15:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75074.log 2023-11-24T09:15:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.74696.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74696.log.gz 2023-11-24T09:15:35.092 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63611.log 2023-11-24T09:15:35.092 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75074.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.75074.log.gz 2023-11-24T09:15:35.093 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68526.log 2023-11-24T09:15:35.095 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63611.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63611.log.gz 2023-11-24T09:15:35.095 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49974.log 2023-11-24T09:15:35.096 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.68526.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68526.log.gz 2023-11-24T09:15:35.097 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52479.log 2023-11-24T09:15:35.098 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.49974.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49974.log.gz 2023-11-24T09:15:35.098 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67063.log 2023-11-24T09:15:35.106 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.52479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70540.log 2023-11-24T09:15:35.107 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67063.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67063.log.gz 2023-11-24T09:15:35.122 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50103.log.gz 2023-11-24T09:15:35.122 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69485.log 2023-11-24T09:15:35.123 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70540.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70540.log.gz 2023-11-24T09:15:35.123 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34778.log 2023-11-24T09:15:35.124 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69485.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69485.log.gz 2023-11-24T09:15:35.125 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54516.log 2023-11-24T09:15:35.125 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.34778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34778.log.gz 2023-11-24T09:15:35.126 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.60320.log 2023-11-24T09:15:35.126 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.54516.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54516.log.gz 2023-11-24T09:15:35.127 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59523.log 2023-11-24T09:15:35.138 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.1.60320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61203.log 2023-11-24T09:15:35.139 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59523.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59523.log.gz 2023-11-24T09:15:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56685.log 2023-11-24T09:15:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61203.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61203.log.gz 2023-11-24T09:15:35.153 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.60320.log.gz 2023-11-24T09:15:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65843.log 2023-11-24T09:15:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56685.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56685.log.gz 2023-11-24T09:15:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69361.log 2023-11-24T09:15:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.65843.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65843.log.gz 2023-11-24T09:15:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61952.log 2023-11-24T09:15:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69361.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69361.log.gz 2023-11-24T09:15:35.157 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40197.log 2023-11-24T09:15:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.61952.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61952.log.gz 2023-11-24T09:15:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40316.log 2023-11-24T09:15:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.40197.log: 55.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40197.log.gz 2023-11-24T09:15:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66877.log 2023-11-24T09:15:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.40316.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40316.log.gz 2023-11-24T09:15:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41557.log 2023-11-24T09:15:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66877.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66877.log.gz 2023-11-24T09:15:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74930.log 2023-11-24T09:15:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.41557.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41557.log.gz 2023-11-24T09:15:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44541.log 2023-11-24T09:15:35.174 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.74930.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41712.log.gz 2023-11-24T09:15:35.178 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78620.log 2023-11-24T09:15:35.184 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.44541.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44541.log.gz 2023-11-24T09:15:35.185 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63415.log 2023-11-24T09:15:35.185 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78620.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78620.log.gz 2023-11-24T09:15:35.191 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52581.log.gz 2023-11-24T09:15:35.194 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76214.log 2023-11-24T09:15:35.195 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.63415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78048.log 2023-11-24T09:15:35.196 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.76214.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63415.log.gz 2023-11-24T09:15:35.196 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.76214.log.gz 2023-11-24T09:15:35.198 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63867.log 2023-11-24T09:15:35.198 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.78048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76989.log 2023-11-24T09:15:35.198 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78048.log.gz 2023-11-24T09:15:35.198 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.63867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59390.log 2023-11-24T09:15:35.198 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76989.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63867.log.gz 2023-11-24T09:15:35.198 INFO:teuthology.orchestra.run.smithi164.stderr: 78.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76989.log.gz 2023-11-24T09:15:35.198 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66793.log 2023-11-24T09:15:35.199 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.59390.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59390.log.gz 2023-11-24T09:15:35.199 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51320.log 2023-11-24T09:15:35.199 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60866.log 2023-11-24T09:15:35.202 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.66793.log: /var/log/ceph/cluster2-client.mirror.51320.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.66793.log.gz 2023-11-24T09:15:35.202 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40214.log 2023-11-24T09:15:35.203 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60866.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60866.log.gz 2023-11-24T09:15:35.203 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50309.log 2023-11-24T09:15:35.204 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.40214.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40214.log.gz 2023-11-24T09:15:35.204 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73588.log 2023-11-24T09:15:35.206 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.50309.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50309.log.gz 2023-11-24T09:15:35.206 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-11-24T09:15:35.207 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.73588.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73588.log.gz 2023-11-24T09:15:35.207 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61102.log 2023-11-24T09:15:35.222 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70968.log 2023-11-24T09:15:35.224 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.61102.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61102.log.gz 2023-11-24T09:15:35.242 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64835.log 2023-11-24T09:15:35.243 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.70968.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70968.log.gz 2023-11-24T09:15:35.263 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.77385.log 2023-11-24T09:15:35.263 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.64835.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64835.log.gz 2023-11-24T09:15:35.271 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71260.log 2023-11-24T09:15:35.271 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.77385.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.77385.log.gz 2023-11-24T09:15:35.278 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60125.log 2023-11-24T09:15:35.279 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.71260.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71260.log.gz 2023-11-24T09:15:35.286 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34802.log 2023-11-24T09:15:35.287 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.60125.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60125.log.gz 2023-11-24T09:15:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79530.log 2023-11-24T09:15:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.34802.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34802.log.gz 2023-11-24T09:15:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53797.log 2023-11-24T09:15:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.admin.79530.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79530.log.gz 2023-11-24T09:15:35.309 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48461.log.gz 2023-11-24T09:15:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51467.log 2023-11-24T09:15:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53797.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53797.log.gz 2023-11-24T09:15:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66414.log 2023-11-24T09:15:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.51467.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51467.log.gz 2023-11-24T09:15:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69836.log 2023-11-24T09:15:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.66414.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66414.log.gz 2023-11-24T09:15:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47886.log 2023-11-24T09:15:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69836.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69836.log.gz 2023-11-24T09:15:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56662.log 2023-11-24T09:15:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.47886.log: 91.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47886.log.gz 2023-11-24T09:15:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58331.log 2023-11-24T09:15:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.56662.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56662.log.gz 2023-11-24T09:15:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42257.log 2023-11-24T09:15:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.58331.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58331.log.gz 2023-11-24T09:15:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42889.log 2023-11-24T09:15:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.42257.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42257.log.gz 2023-11-24T09:15:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67877.log 2023-11-24T09:15:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.42889.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42889.log.gz 2023-11-24T09:15:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76717.log 2023-11-24T09:15:35.333 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.67877.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.67877.log.gz 2023-11-24T09:15:35.334 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53777.log 2023-11-24T09:15:35.335 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.76717.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76717.log.gz 2023-11-24T09:15:35.335 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59008.log 2023-11-24T09:15:35.337 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.53777.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53777.log.gz 2023-11-24T09:15:35.337 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69464.log 2023-11-24T09:15:35.338 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.59008.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59008.log.gz 2023-11-24T09:15:35.339 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75791.log 2023-11-24T09:15:35.341 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.69464.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69464.log.gz 2023-11-24T09:15:35.342 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53733.log 2023-11-24T09:15:35.344 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.75791.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75791.log.gz 2023-11-24T09:15:35.344 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.44650.log 2023-11-24T09:15:35.345 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.53733.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53733.log.gz 2023-11-24T09:15:35.345 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42734.log.gz 2023-11-24T09:15:35.345 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41736.log 2023-11-24T09:15:35.354 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster2-client.mirror.0.44650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47931.log 2023-11-24T09:15:35.355 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.admin.41736.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41736.log.gz 2023-11-24T09:15:35.356 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cluster1-client.mirror.47931.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47931.log.gz 2023-11-24T09:15:35.363 INFO:teuthology.orchestra.run.smithi164.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74930.log.gz 2023-11-24T09:15:35.411 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52479.log.gz 2023-11-24T09:15:35.468 INFO:teuthology.orchestra.run.smithi164.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.44650.log.gz 2023-11-24T09:15:35.529 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51320.log.gz 2023-11-24T09:15:39.848 INFO:teuthology.orchestra.run.smithi153.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-11-24T09:15:40.936 INFO:teuthology.orchestra.run.smithi153.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-11-24T09:15:40.990 INFO:teuthology.orchestra.run.smithi153.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-11-24T09:15:40.991 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-11-24T09:15:40.991 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m9.804s 2023-11-24T09:15:40.991 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m28.943s 2023-11-24T09:15:40.991 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m1.182s 2023-11-24T09:16:17.989 INFO:teuthology.orchestra.run.smithi164.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-11-24T09:16:20.528 INFO:teuthology.orchestra.run.smithi164.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-11-24T09:16:26.375 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-11-24T09:16:26.377 INFO:teuthology.orchestra.run.smithi164.stderr: 2023-11-24T09:16:26.377 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m55.156s 2023-11-24T09:16:26.377 INFO:teuthology.orchestra.run.smithi164.stderr:user 2m44.586s 2023-11-24T09:16:26.377 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m6.534s 2023-11-24T09:16:26.377 INFO:tasks.ceph:Archiving logs... 2023-11-24T09:16:26.378 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501/remote/smithi153/log 2023-11-24T09:16:26.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-24T09:16:29.729 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501/remote/smithi164/log 2023-11-24T09:16:29.730 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-24T09:16:50.268 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-24T09:16:50.285 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2771, in wait_for_clean cur_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_135a1ac939fb4e6d1ef988cf89d0da570e04d3d1/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi164 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-24T09:16:50.286 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-24T09:16:50.286 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-24T09:16:50.289 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-24T09:16:50.349 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-24T09:16:50.360 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-24T09:16:50.361 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T09:16:50.363 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T09:16:50.374 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2023-11-24T09:16:50.374 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2023-11-24T09:16:50.374 INFO:teuthology.orchestra.run.smithi153.stdout:*hv01.front.sepi 44.4.53.2 2 u 5 64 377 0.086 -2.108 0.275 2023-11-24T09:16:50.374 INFO:teuthology.orchestra.run.smithi153.stdout:+hv02.front.sepi 96.245.170.99 3 u 55 64 377 0.093 -0.215 0.240 2023-11-24T09:16:50.374 INFO:teuthology.orchestra.run.smithi153.stdout:+hv03.front.sepi 74.6.168.72 3 u 2 64 377 0.089 -1.029 0.137 2023-11-24T09:16:50.374 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-11-24T09:16:50.376 INFO:teuthology.orchestra.run.smithi164.stdout: remote refid st t when poll reach delay offset jitter 2023-11-24T09:16:50.376 INFO:teuthology.orchestra.run.smithi164.stdout:============================================================================== 2023-11-24T09:16:50.376 INFO:teuthology.orchestra.run.smithi164.stdout:*hv01.front.sepi 44.4.53.2 2 u 43 64 377 0.088 -2.281 0.487 2023-11-24T09:16:50.376 INFO:teuthology.orchestra.run.smithi164.stdout:+hv02.front.sepi 96.245.170.99 3 u 45 64 377 0.086 -0.878 0.293 2023-11-24T09:16:50.376 INFO:teuthology.orchestra.run.smithi164.stdout:+hv03.front.sepi 74.6.168.72 3 u 61 64 377 0.094 -1.642 0.206 2023-11-24T09:16:50.376 INFO:teuthology.orchestra.run.smithi164.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-11-24T09:16:50.377 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-24T09:16:50.388 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-24T09:16:50.389 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-24T09:16:50.402 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-24T09:16:50.413 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-24T09:16:50.425 INFO:teuthology.task.internal:Duration was 3484.626037 seconds 2023-11-24T09:16:50.425 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-24T09:16:50.437 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-24T09:16:50.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-24T09:16:50.439 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-24T09:16:50.518 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-24T09:16:50.518 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2023-11-24T09:16:50.518 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-24T09:16:50.528 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2023-11-24T09:16:50.528 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-24T09:16:50.538 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-24T09:16:50.538 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-24T09:16:50.573 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-24T09:16:50.604 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-24T09:16:50.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-24T09:16:50.637 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-24T09:16:50.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-24T09:16:50.714 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-24T09:16:50.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-24T09:16:50.727 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-24T09:16:50.739 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-24T09:16:50.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-24T09:16:50.769 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-24T09:16:50.778 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2023-11-24T09:16:50.792 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2023-11-24T09:16:50.804 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-24T09:16:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T09:16:50.848 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-24T09:16:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T09:16:50.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-24T09:16:50.866 INFO:teuthology.task.internal:Transferring archived files... 2023-11-24T09:16:50.867 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501/remote/smithi153 2023-11-24T09:16:50.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-24T09:16:50.909 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501/remote/smithi164 2023-11-24T09:16:50.909 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-24T09:16:50.928 INFO:teuthology.task.internal:Removing archive directory... 2023-11-24T09:16:50.929 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-24T09:16:50.950 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-24T09:16:50.971 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-24T09:16:50.986 INFO:teuthology.task.internal:Not uploading archives. 2023-11-24T09:16:50.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-24T09:16:50.999 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-24T09:16:50.999 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-24T09:16:51.001 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-24T09:16:51.004 INFO:teuthology.orchestra.run.smithi153.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 24 09:16 /home/ubuntu/cephtest 2023-11-24T09:16:51.019 INFO:teuthology.orchestra.run.smithi164.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 24 09:16 /home/ubuntu/cephtest 2023-11-24T09:16:51.021 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-24T09:16:51.034 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-24T09:16:51.048 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi164 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-24T09:16:51.048 INFO:teuthology.nuke:Checking targets against current locks 2023-11-24T09:16:51.077 DEBUG:teuthology.nuke:shortname: smithi153 2023-11-24T09:16:51.077 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T09:16:51.105 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 08:10:02.400326', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKay/u69gbSDH+wQg1rANAqFcLSHi3oa5yrvt7QTzuCp6Tnk1XI0S9vKeGRUUVuR7yn7Ern95TqUza/7zUGQSdo='} 2023-11-24T09:16:51.109 DEBUG:teuthology.nuke:shortname: smithi164 2023-11-24T09:16:51.109 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T09:16:51.129 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-11-24_05:31:39-rbd-wip-dis-testing-distro-default-smithi/7465501', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 08:10:02.399211', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNN4flvWtU6IA2MNQJD2lLFhHIgd/PZ0iehpJcUsE7kQw/8ZZUYwpfu5sODRLPlVm8TccYiZoQyDcPZAMHRMrk='} 2023-11-24T09:16:51.158 INFO:teuthology.orchestra.console.smithi153:Power off 2023-11-24T09:16:51.158 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-24T09:16:51.181 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2023-11-24T09:16:51.182 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T09:16:51.199 INFO:teuthology.orchestra.console.smithi164:Power off 2023-11-24T09:16:51.199 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-24T09:16:51.208 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-11-24T09:16:51.223 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2023-11-24T09:16:51.223 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T09:16:51.246 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2023-11-24T09:16:55.209 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T09:16:55.248 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T09:16:55.319 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-11-24T09:16:55.357 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2023-11-24T09:16:59.321 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T09:16:59.358 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T09:16:59.430 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2023-11-24T09:16:59.468 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2023-11-24T09:16:59.531 INFO:teuthology.orchestra.console.smithi153:Power off completed 2023-11-24T09:16:59.569 INFO:teuthology.orchestra.console.smithi164:Power off completed 2023-11-24T09:16:59.673 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 3484.6260373592377 failure_reason: 'Command failed on smithi164 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json''' flavor: default owner: scheduled_dis@teuthology status: fail success: false 2023-11-24T09:16:59.673 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-24T09:16:59.762 INFO:teuthology.run:FAIL